Introduction: Virtual Reality Teleconferencing Device With Google Cardboard

Virtual Reality Teleconferencing Device with Google Cardboard

Created by John Choi | 2015

This project is a video teleconferencing device with two webcams for stereoscopic vision placed on top of a 3-axis motorized platform, controlled by a head-tracking head mounted display.

Note that virtual reality in general is still a technology in development, and there are many areas for improvement. Specifically, there is a noticable delay between head rotation and computer response, and sending video and angle information over a network only exacerbates this issue. Also, the servo motors used in this project occasionally "jitter," causing the device to rotate in small increments without input. Ultimately, the device demonstrated here is very finicky at best and should not be considered a completed product in terms of either reliability or usability.

In order to build and run this project, you will need rudimentary electronics skill, access to 3D printing, and the ability to use a personal computer (Windows 7 or above). Estimated time for construction is one week, and expected cost for this project is roughly 60 USD (variable depending on what parts you already have).

Step 1: Parts & Software

You will need the following parts for this project: (Sample links provided. Substitutes may be used.)

You will need the following software for this project:

Step 2: 3D Printing

Go to this GitHub repository: (https://github.com/johnchoi313/Virtual-Reality-Teleconferencing-Device)

Click on the download ZIP button, and extract the downloaded files. Navigate to the CAD folder. Here, you will find all necessary files to 3D print in STL format. Go ahead and 3D print all 4 parts in whatever colors you please.

If you do not have a 3D printer, I recommend using a 3D printing service like 3D Hubs to find a local 3D printer in your area. Expected print time for all parts is around 4 hours.

Step 3: Assembly

There are 3 servo motors to attach to 4 3D printed parts. This can be done in the following steps:

  1. Attach the servo motors to Parts 1, 2, and 3 as shown. Make sure the orientation of the servo is correct! Secure the servo motors by using the 4-40 screws.
  2. Attach the the servo horns to the servo motors at their 90 positions. You can make sure the horns are attached in the correct rotation by rotating the motor 90 degrees forward and backward.
  3. Attach the Part 1/servo piece to the bottom of the Part 2/servo piece. Secure the connection in place using two 2-56 screws.
  4. Attach the Part 2/servo piece onto the U-shaped portion of the Part 3/servo piece. Use three 2-56 screws on the servo horn side of the Part 2/servo piece and one 4-40 screw on the other side.
  5. Attach Part 4 to the servo horn on the top of the Part 3/servo piece.
  6. Attach the Mini Camera Tripod to the bottom of the Part 1/servo piece.

Note: Due to the nature of 3D printing, it is possible that the screws may not fit. If this is the case, use a drill to make the holes larger until the screws do fit.

Step 4: Electronics

An Arduino UNO is used to control the 3 servo motors, which are powered with 4 standard AA batteries.

If you've had any experience with RC servomotors, you'll know there are 3 pins for each servo: red/orange for voltage input, black/brown for ground, and white/yellow for signal. Because 3 standard servo motors drives more current than the Arduino UNO is able to handle, we will need a separate power source (in this case the 4 AA batteries). Also, as this device has 3 degrees of freedom, there are 3 digital pins to consider: one to control the base servo (Yaw), one to control the middle servo (Pitch), and one to control the top servo (Roll).

Create the electronic wiring as shown in the Fritzing Diagram above. Note that the servos are wired in parallel.

Step 5: Uploading Arduino Code

This project runs on the Standard Firmata protocol. In essence, Standard Firmata is an easy to use interface with the Arduino board that allows continuous real-time control through an application on another device, such as Processing on a laptop. Do the following steps to setup Standard Firmata:

  1. Open the Arduino IDE and connect the Arduino UNO to your computer using the USB A to B cable.
  2. Click on File->Examples->Firmata->StandardFirmata. This will load the necessary example code onto the Arduino IDE.
  3. Now, click on Tools->Board->Arduino UNO. This will change the board type so that the IDE knows what hardware it is working with.
  4. Finally, click on Tools->Serial Port->(Serial Port Name). This will set the Arduino IDE to the serial port where the Arduino is connected (usually, the Serial Port Name is something like COM# on Windows).
  5. Upload the Standard Firmata code onto the Arduino by clicking on the arrow button next to the check mark on the upper left of the Arduino IDE. If there are no glaring orange errors in the Arduino IDE log, then the code has been uploaded successfully onto the Arduino board.

You can test the Standard Firmata protocol and make sure it works by using the Firmata Test application. You can download this here. Run the Firmata_Test executable, and click on Port->(Serial Port Name). The Serial Port Name is the same one used to upload the code in the Arduino IDE. After a few seconds, you should see a lot of pin information appear, which can be controlled in real time. In particular, we are interested in pins 8, 9 and 10. Change their type to "Servo" and adjust their sliders to see that they work as expected.

Step 6: Setting Up TrinusVR for Google Cardboard

The modern virtual reality device is essentially a head-mounted screen that displays stereoscopic visuals while tracking the user's head rotation in real-time. The continuous feedback between the user's head movements and the stereoscopic contents of the display generates an illusion of being in another place, hence the name "virtual reality."

For our purposes, we will be using an Android smart phone locked inside a Google Cardboard to generate both the stereoscopic display and the gyroscopic head tracking to produce the illusion of virtual reality.

Because our project is primarily PC based, we use TrinusVR to stream video and head-tracking data from the Android smart phone to the main computer. Note that you will need TrinusVR installed on both your phone and your computer for this to work. More documentation on how to setup and use TrinusVR can be found here.

Note: It is possible to use another virtual reality device such as the Oculus Rift. It will be compatible with the Virtual Reality Teleconferencing Device as long as the head-tracking data can be used to control the computer's mouse movements.

Step 7: Running the VRTD Software

This project uses custom software written in Processing 2.0 to remotely control the Virtual Reality Teleconferencing Device (This software can be found in the VRTD link in Step 1). This software is separated into two parts: VRTD_sender and VRTD_receiver.

In summary, VRTD_sender is used on the computer connected to the virtual reality device. This software tracks mouse movements generated by head tracking, and streams it to a specified IP address. This IP address belongs to a second computer, which is in a remote location running VRTD_receiver. VRTD_receiver stereoscopically displays video data from the left and right webcams, and also controls the rotation of the Virtual Reality Teleconferencing Device according to the data sent from VRTD_sender. The stereoscopic webcam data can then be streamed through a video chat program like Skype with screensharing enabled.

To do Virtual Reality Teleconferencing, do the following steps:

  1. Have 2 Windows computers ready (Computer A and Computer B).
  2. Hook up Computer A with the Google Cardboard, and run VRTD_sender.exe.
  3. Hook up Computer B with the Virtual Reality Teleconferencing Device, and run VRTD_receiver.exe.
  4. Set the Left and Right Webcams and the Arduino Serial Port in VRTD_receiver.
  5. Run Skype on both computers and initiate a video call with screen sharing.
  6. Enter the IP Address of Computer B in VRTD_sender.
  7. Wear the virtual reality display, which controls the mouse, and hence the motors on Computer B.

Note: It is possible to use the Virtual Reality Teleconferencing system on one computer. If you wish to do this, run VRTD_sender and VRTD_receiver on the same machine and use 127.0.0.1 as the target IP address.

Step 8: BONUS - Improved Control Software

In the previous step, instructions to setup software were given with VRTD Sender and VRTD Receiver, powered by Processing. However, some limitations must be considered in the fact that some of the parts of the original system were proprietary, work only on Windows, and have tremendous lag.

To address these issues, an awesome user by the name of volkovmqx decided to make his own software to make the Virtual Reality Teleconferencing Device work on any platform (Windows, Mac, and Linux), and because he's awesome, he's released it for free on GitHub. Some details about the new and improved software:

  1. The Streaming server /StreamingServer that uses CVLC and HTTP MJPEG.
  2. The Position Server /motionServer that grabes REST Requests from the Phone Gyroscope and Send them to the Arduino Through Serial. Technologies used : Python, Tornado and PySerial.
  3. VR ready Mobile App /mobileApp , made by Ionic so it can be used on Iphone or Android, that consume the two Streams and send the Gyroscope data to the WebServer Created by /motionServer .
  4. Arduino Code /Arduino to move the Servos to the desired place.

The video above shows the new software in action with a plastic Cardboard.

Step 9: Conclusion

Congratulations! If you got to this page while building everything along the way, you should now have a fully functional Virtual Reality Teleconferencing system. You can now use it to immerse yourself in places across the world without actually having to be there.

This project is part of the Phone Contest, so if you liked it, please be sure to press the "Vote" button on the upper-right hand corner on this page.

If you wish to expand upon the Virtual Reality Teleconferencing Device, feel free to do so! All the necessary CAD files and Processing source code made specifically for this project have been released under a free Open Source License.

Happy Virtual Reality Teleconferencing!

-John C.

Video Intro Music Credits:

"Sancho Panza Gets a Latte" by Kevin Macleod (incompetech.com)

Licensed under Creative Commons: By Attribution 3.0 http://creativecommons.org/licenses/by/3.0/

Comments

author
volkovmqx made it! (author)2016-05-16

Hello, I Made this so it can run on any device, with better performance. (Mobile App that works on Android and Iphone, Stream App that works on all plateforms and guess what? The source code is free to hack :D ) Check this out : https://github.com/volkovmqx/VRS

Screenshot from 2016-05-17 04:56:27.png
author
flavorfan (author)volkovmqx2016-07-22

your code can not work,mobile app miss some file.

author
volkovmqx (author)flavorfan2016-07-22

i think you should install ionic, and do npm install & bower install. if not i will be happy to answer you @ github. please fill an issue there. thanks !

author
EvanderE (author)volkovmqx2016-06-19

Can I have some instructions on how to use this? hahaha im a noob

author
volkovmqx (author)EvanderE2016-06-19

https://github.com/volkovmqx/VRS have the documentation (search on how to use each technology : ionic, python etc)

author
EvanderE (author)volkovmqx2016-06-19

Thank you so much sir :)

author
John Choi (author)volkovmqx2016-05-17

Wow, this looks awesome, thanks for sharing! I'll be sure to check it out, and by the looks of it, your software seems far better than mine :D

May I include a link to your work as part of this Instructable?

author
volkovmqx (author)John Choi2016-05-17

Thanks :D
Yes ! i will be happy that you will include this as a part of your instructables.

author
John Choi (author)volkovmqx2016-05-18

The Instructable has been updated. See Step 8 for inclusion of your improved control software. Thanks again and have a wonderful day :)

author
volkovmqx (author)John Choi2016-05-18

Awesome ! Thanks.

author
blacx13 (author)2016-01-30

hello , im trying to use this over the internet , the servo works completely fine on local area or within the router , but over the internet doesnt seem to work using public ip address any suggestion ?

author
John Choi (author)blacx132016-01-30

Hm, I'm actually not sure on this one.

As you may have noticed, this is powered by the Open Sound Control protocol (which I did not write), and specifically the OSC library for Processing. From my understanding, if you got it to work within the local area network, it should also work over the internet as well if you got the correct IP address. Some more information on it is here: http://www.sojamo.de/libraries/oscP5/#features

And here: https://github.com/sojamo/oscp5

If you are lucky, you may be able to contact the person that wrote the OSC library :)

author
blacx13 (author)John Choi2016-02-08

hi, how can i see the webcam output from the receiver if i should focus on the sender.exe to move the servo motors . im using 2 computers

author
John Choi (author)blacx132016-02-08

You can use a video chat program like Skype to send the video data from the receiver on the first computer to the second computer. You then use TrinusVR on the second computer to transmit the Skype window onto your Google Cardboard device. (I know is this somewhat convoluted, but it does sort of work.)

author
blacx13 (author)John Choi2016-02-09

yeah , the video would be transmitted but does the servo would still work if i focused on skype window on the second computer with sender

author
John Choi (author)blacx132016-02-09

Skype has a screenshare function that runs in the background, allowing you to focus on the sender application.

author
EvanderE (author)2015-11-22

Umm I'm having trouble in selecting the servos...what servos are you using? there are turning point of (90) (180) (360 continuous rotation) ... I don't know what to choose...I'm about to buy TowerPro MG-995. Thanks in Advance

author
EvanderE (author)John Choi2016-01-11

Hi, I Finally Have all materials needed. I followed all the instructions and it worked! .. I'm using only one computer. I have a problem. I could only control the servos when I'm with the "VRTD Sender" tab so to be able to see the visuals I have to switch with "VRTD Receiver" but when I'm in this tab I can't move the servos anymore.. I could only see "Gray" when I'm with the "VRTD Sender" tab Did I miss something?.. hehehe I suddenly get a burst of happy feeling when I got the servos to work lol

author
John Choi (author)EvanderE2016-01-12

Great! Yes, that is correct - you have to be focused on the VRTD Sender tab for the servos to move. In an ideal situation, VRTD Sender and VRTD Receiver are on different computers, allowing you to see both screens at the same time while controlling the servos.

author
815669856 (author)2015-11-19

the VRTD receiver could control the servo through i update the xRot,yRot,zRot.but the VRTD receiver couldn't receive the x,y,z from VRTD sender.

author
John Choi (author)8156698562015-11-19

I am assuming you are using one computer to test both the VRTD receiver and sender simultaneously (this is fine). Try checking which window you have focused. You should have VRTD sender as the window up front and VRTD receiver in the background, otherwise VRTD sender will not register the mouse position.

author
815669856 (author)John Choi2015-11-19

At the present, I could control the servo with mouse moving.

But, the driver of two different camera are the same when i bought them. Then I I amend one of them to other driver .The VRTD receive only can recognize one of them.

should i update the other different driver to camera or other setting?

Thanks for helping me to debug the device.

author
John Choi (author)8156698562015-11-20

Yes, you need two different webcams for this to work, as Windows cannot differentiate between two of the same webcam.

author
815669856 (author)John Choi2015-11-19

At the present, I could control the servo with mouse moving.

But, the driver of two different camera are the same when i bought them. Then I I amend one of them to other driver .The VRTD receive only can recognize one of them.

should i update the other different driver to camera or other setting?

Thanks for helping me to debug the device.

author
815669856 (author)8156698562015-11-19

now, I connect phone to computer through wifi.

author
John Choi (author)8156698562015-11-19

I have another possible fix in another reply.
(Also, if possible, please comment on the Instructables page so that everyone can see potential fixes.)

author
815669856 made it! (author)2015-11-19

now,i could run firmata_test.exe to test servo,the servo could run following i hoped.but when i run both the VRTD receiver and sender ,and set IP to 127.0.0.1.The servo couldn't rotate following mouse.

I set all the parameters that you set.Do I need to other Settings?

QQ截图20151119163120.pngQQ截图20151119163159.png
author
John Choi (author)8156698562015-11-19

I have another possible fix in another reply.
(Also, if possible, please comment on the Instructables page so that everyone can see potential fixes.)

author
815669856 made it! (author)2015-11-19

should i set IP or port?

QQ截图20151119222311.png
author
John Choi (author)8156698562015-11-19

Since Trinus is able to control the mouse, that means it has been configured correctly. We just need to configure VRTD receiver and VRTD sender (addressed in another reply).

author
Luca Steinmetz (author)2015-11-11

I have a problem, when i run VRTD_sender or reciver, comes out an error pop up: C:\ProgramData\Oracle\Java\Javapath\javaw.exe

author
John Choi (author)Luca Steinmetz2015-11-11

Hm, that's odd. What version of Windows and Processing are you using?

author
Luca Steinmetz (author)John Choi2015-11-12

I solved that problem. I creat my own processing program and now it' s working.

but I have another question: how can i find the correct position of the two webcam?

author
John Choi (author)Luca Steinmetz2015-11-12

Great! You learn a lot ore by doing it yourself :)

The best way to find the correct position of the webcam is to measure the distance between your eyes and space the webcams the same distance.

You can also put them roughly spaced at eye length and rotate them until they "feel right" through the VR headset.

author
Luca Steinmetz (author)John Choi2015-11-12

ok, I will try.

author
John Choi (author)Luca Steinmetz2015-11-12

"Do. Or do not. There is no try." -Yoda

author
815669856 made it! (author)2015-11-09

your code is full?

123.png
author
815669856 (author)8156698562015-11-09

I open this code in processing

author
John Choi (author)8156698562015-11-09

Have you installed the oscP5 library first? If not, try installing it, following this documentation: https://github.com/processing/processing/wiki/How-...

You can download the oscP5 library here: http://www.sojamo.de/libraries/oscP5/

author
815669856 (author)John Choi2015-11-10

at current situation,I run VRTD_receiver.exe.but i can't see anything.this is the reason that i open the code in processing.

author
John Choi (author)8156698562015-11-11

Problem fixed by using Processing version 2.2.1. Have a nice day :)

author
815669856 made it! (author)John Choi2015-11-09

thanks for your replying,but i have a question for the code.The problem about oscP5 have been worked out. I don't know where i can find the Capture library.can you give me some sugguestion?(I am a newer.Thank you for your patience to answer my question)

oujie.png
author
John Choi (author)8156698562015-11-10

Capture should be one of Processing's default libraries. See documentation here: https://processing.org/reference/libraries/video/Capture.html. Which version of Processing are you using?

author
815669856 made it! (author)8156698562015-11-10

how to install the library to processing.this library is downloaded by me on the internet. if i don't connect the camera,can the VRTD_receiver.exe run fine ?

oujie.png
author
John Choi (author)8156698562015-11-10

The VRTD receiver is receiving the mouse data from the VRTD sender, so I believe it should still work without the webcam output.

author
815669856 (author)2015-11-09

that's a interesting application .but why i can't run VRTD_receiver.exe? do i require other setting?

author
John Choi (author)8156698562015-11-09

Assuming you are using a Windows 7 computer, I think it should run just fine. What is the error you are receiving?

author
dombengi (author)2015-09-30

what exactly am I looking at?

author
John Choi (author)dombengi2015-10-01

Depends on what you see.

It is a virtual reality teleconferencing device so that you can immerse yourself in other locations without actually being there. That being said, I recognize the device looks a bit like a three-legged alien ;)

About This Instructable

10,286views

156favorites

License:

Bio: Just a guy that likes building robots and videogames.
More by John Choi:Multipurpose Mobile Manipulator MkIIMultipurpose Mobile Manipulator Mk 1Virtual Reality Teleconferencing Device with Google Cardboard
Add instructable to: