Virtual Reality Teleconferencing Device With Google Cardboard

11,493

159

36

About: Just a guy that likes building robots and videogames.

Virtual Reality Teleconferencing Device with Google Cardboard

Created by John Choi | 2015

This project is a video teleconferencing device with two webcams for stereoscopic vision placed on top of a 3-axis motorized platform, controlled by a head-tracking head mounted display.

Note that virtual reality in general is still a technology in development, and there are many areas for improvement. Specifically, there is a noticable delay between head rotation and computer response, and sending video and angle information over a network only exacerbates this issue. Also, the servo motors used in this project occasionally "jitter," causing the device to rotate in small increments without input. Ultimately, the device demonstrated here is very finicky at best and should not be considered a completed product in terms of either reliability or usability.

In order to build and run this project, you will need rudimentary electronics skill, access to 3D printing, and the ability to use a personal computer (Windows 7 or above). Estimated time for construction is one week, and expected cost for this project is roughly 60 USD (variable depending on what parts you already have).

Step 1: Parts & Software

You will need the following parts for this project: (Sample links provided. Substitutes may be used.)

You will need the following software for this project:

Step 2: 3D Printing

Go to this GitHub repository: (https://github.com/johnchoi313/Virtual-Reality-Teleconferencing-Device)

Click on the download ZIP button, and extract the downloaded files. Navigate to the CAD folder. Here, you will find all necessary files to 3D print in STL format. Go ahead and 3D print all 4 parts in whatever colors you please.

If you do not have a 3D printer, I recommend using a 3D printing service like 3D Hubs to find a local 3D printer in your area. Expected print time for all parts is around 4 hours.

Step 3: Assembly

There are 3 servo motors to attach to 4 3D printed parts. This can be done in the following steps:

  1. Attach the servo motors to Parts 1, 2, and 3 as shown. Make sure the orientation of the servo is correct! Secure the servo motors by using the 4-40 screws.
  2. Attach the the servo horns to the servo motors at their 90 positions. You can make sure the horns are attached in the correct rotation by rotating the motor 90 degrees forward and backward.
  3. Attach the Part 1/servo piece to the bottom of the Part 2/servo piece. Secure the connection in place using two 2-56 screws.
  4. Attach the Part 2/servo piece onto the U-shaped portion of the Part 3/servo piece. Use three 2-56 screws on the servo horn side of the Part 2/servo piece and one 4-40 screw on the other side.
  5. Attach Part 4 to the servo horn on the top of the Part 3/servo piece.
  6. Attach the Mini Camera Tripod to the bottom of the Part 1/servo piece.

Note: Due to the nature of 3D printing, it is possible that the screws may not fit. If this is the case, use a drill to make the holes larger until the screws do fit.

Step 4: Electronics

An Arduino UNO is used to control the 3 servo motors, which are powered with 4 standard AA batteries.

If you've had any experience with RC servomotors, you'll know there are 3 pins for each servo: red/orange for voltage input, black/brown for ground, and white/yellow for signal. Because 3 standard servo motors drives more current than the Arduino UNO is able to handle, we will need a separate power source (in this case the 4 AA batteries). Also, as this device has 3 degrees of freedom, there are 3 digital pins to consider: one to control the base servo (Yaw), one to control the middle servo (Pitch), and one to control the top servo (Roll).

Create the electronic wiring as shown in the Fritzing Diagram above. Note that the servos are wired in parallel.

Step 5: Uploading Arduino Code

This project runs on the Standard Firmata protocol. In essence, Standard Firmata is an easy to use interface with the Arduino board that allows continuous real-time control through an application on another device, such as Processing on a laptop. Do the following steps to setup Standard Firmata:

  1. Open the Arduino IDE and connect the Arduino UNO to your computer using the USB A to B cable.
  2. Click on File->Examples->Firmata->StandardFirmata. This will load the necessary example code onto the Arduino IDE.
  3. Now, click on Tools->Board->Arduino UNO. This will change the board type so that the IDE knows what hardware it is working with.
  4. Finally, click on Tools->Serial Port->(Serial Port Name). This will set the Arduino IDE to the serial port where the Arduino is connected (usually, the Serial Port Name is something like COM# on Windows).
  5. Upload the Standard Firmata code onto the Arduino by clicking on the arrow button next to the check mark on the upper left of the Arduino IDE. If there are no glaring orange errors in the Arduino IDE log, then the code has been uploaded successfully onto the Arduino board.

You can test the Standard Firmata protocol and make sure it works by using the Firmata Test application. You can download this here. Run the Firmata_Test executable, and click on Port->(Serial Port Name). The Serial Port Name is the same one used to upload the code in the Arduino IDE. After a few seconds, you should see a lot of pin information appear, which can be controlled in real time. In particular, we are interested in pins 8, 9 and 10. Change their type to "Servo" and adjust their sliders to see that they work as expected.

Step 6: Setting Up TrinusVR for Google Cardboard

The modern virtual reality device is essentially a head-mounted screen that displays stereoscopic visuals while tracking the user's head rotation in real-time. The continuous feedback between the user's head movements and the stereoscopic contents of the display generates an illusion of being in another place, hence the name "virtual reality."

For our purposes, we will be using an Android smart phone locked inside a Google Cardboard to generate both the stereoscopic display and the gyroscopic head tracking to produce the illusion of virtual reality.

Because our project is primarily PC based, we use TrinusVR to stream video and head-tracking data from the Android smart phone to the main computer. Note that you will need TrinusVR installed on both your phone and your computer for this to work. More documentation on how to setup and use TrinusVR can be found here.

Note: It is possible to use another virtual reality device such as the Oculus Rift. It will be compatible with the Virtual Reality Teleconferencing Device as long as the head-tracking data can be used to control the computer's mouse movements.

Step 7: Running the VRTD Software

This project uses custom software written in Processing 2.0 to remotely control the Virtual Reality Teleconferencing Device (This software can be found in the VRTD link in Step 1). This software is separated into two parts: VRTD_sender and VRTD_receiver.

In summary, VRTD_sender is used on the computer connected to the virtual reality device. This software tracks mouse movements generated by head tracking, and streams it to a specified IP address. This IP address belongs to a second computer, which is in a remote location running VRTD_receiver. VRTD_receiver stereoscopically displays video data from the left and right webcams, and also controls the rotation of the Virtual Reality Teleconferencing Device according to the data sent from VRTD_sender. The stereoscopic webcam data can then be streamed through a video chat program like Skype with screensharing enabled.

To do Virtual Reality Teleconferencing, do the following steps:

  1. Have 2 Windows computers ready (Computer A and Computer B).
  2. Hook up Computer A with the Google Cardboard, and run VRTD_sender.exe.
  3. Hook up Computer B with the Virtual Reality Teleconferencing Device, and run VRTD_receiver.exe.
  4. Set the Left and Right Webcams and the Arduino Serial Port in VRTD_receiver.
  5. Run Skype on both computers and initiate a video call with screen sharing.
  6. Enter the IP Address of Computer B in VRTD_sender.
  7. Wear the virtual reality display, which controls the mouse, and hence the motors on Computer B.

Note: It is possible to use the Virtual Reality Teleconferencing system on one computer. If you wish to do this, run VRTD_sender and VRTD_receiver on the same machine and use 127.0.0.1 as the target IP address.

Step 8: BONUS - Improved Control Software

In the previous step, instructions to setup software were given with VRTD Sender and VRTD Receiver, powered by Processing. However, some limitations must be considered in the fact that some of the parts of the original system were proprietary, work only on Windows, and have tremendous lag.

To address these issues, an awesome user by the name of volkovmqx decided to make his own software to make the Virtual Reality Teleconferencing Device work on any platform (Windows, Mac, and Linux), and because he's awesome, he's released it for free on GitHub. Some details about the new and improved software:

  1. The Streaming server /StreamingServer that uses CVLC and HTTP MJPEG.
  2. The Position Server /motionServer that grabes REST Requests from the Phone Gyroscope and Send them to the Arduino Through Serial. Technologies used : Python, Tornado and PySerial.
  3. VR ready Mobile App /mobileApp , made by Ionic so it can be used on Iphone or Android, that consume the two Streams and send the Gyroscope data to the WebServer Created by /motionServer .
  4. Arduino Code /Arduino to move the Servos to the desired place.

The video above shows the new software in action with a plastic Cardboard.

Step 9: Conclusion

Congratulations! If you got to this page while building everything along the way, you should now have a fully functional Virtual Reality Teleconferencing system. You can now use it to immerse yourself in places across the world without actually having to be there.

This project is part of the Phone Contest, so if you liked it, please be sure to press the "Vote" button on the upper-right hand corner on this page.

If you wish to expand upon the Virtual Reality Teleconferencing Device, feel free to do so! All the necessary CAD files and Processing source code made specifically for this project have been released under a free Open Source License.

Happy Virtual Reality Teleconferencing!

-John C.

Video Intro Music Credits:

"Sancho Panza Gets a Latte" by Kevin Macleod (incompetech.com)

Licensed under Creative Commons: By Attribution 3.0 http://creativecommons.org/licenses/by/3.0/

Phone Contest

Second Prize in the
Phone Contest

6 People Made This Project!

Recommendations

  • Big and Small Contest

    Big and Small Contest
  • Make it Glow Contest 2018

    Make it Glow Contest 2018
  • First Time Author

    First Time Author

36 Discussions

0
None
blacx13

2 years ago

hello , im trying to use this over the internet , the servo works completely fine on local area or within the router , but over the internet doesnt seem to work using public ip address any suggestion ?

5 replies
0
None
John Choiblacx13

Reply 2 years ago

Hm, I'm actually not sure on this one.

As you may have noticed, this is powered by the Open Sound Control protocol (which I did not write), and specifically the OSC library for Processing. From my understanding, if you got it to work within the local area network, it should also work over the internet as well if you got the correct IP address. Some more information on it is here: http://www.sojamo.de/libraries/oscP5/#features

And here: https://github.com/sojamo/oscp5

If you are lucky, you may be able to contact the person that wrote the OSC library :)

0
None
blacx13John Choi

Reply 2 years ago

hi, how can i see the webcam output from the receiver if i should focus on the sender.exe to move the servo motors . im using 2 computers

0
None
John Choiblacx13

Reply 2 years ago

You can use a video chat program like Skype to send the video data from the receiver on the first computer to the second computer. You then use TrinusVR on the second computer to transmit the Skype window onto your Google Cardboard device. (I know is this somewhat convoluted, but it does sort of work.)

0
None
blacx13John Choi

Reply 2 years ago

yeah , the video would be transmitted but does the servo would still work if i focused on skype window on the second computer with sender

0
None
John Choiblacx13

Reply 2 years ago

Skype has a screenshare function that runs in the background, allowing you to focus on the sender application.

0
None
EvanderE

3 years ago

Umm I'm having trouble in selecting the servos...what servos are you using? there are turning point of (90) (180) (360 continuous rotation) ... I don't know what to choose...I'm about to buy TowerPro MG-995. Thanks in Advance

3 replies
0
None
EvanderEJohn Choi

Reply 2 years ago

Hi, I Finally Have all materials needed. I followed all the instructions and it worked! .. I'm using only one computer. I have a problem. I could only control the servos when I'm with the "VRTD Sender" tab so to be able to see the visuals I have to switch with "VRTD Receiver" but when I'm in this tab I can't move the servos anymore.. I could only see "Gray" when I'm with the "VRTD Sender" tab Did I miss something?.. hehehe I suddenly get a burst of happy feeling when I got the servos to work lol

0
None
John ChoiEvanderE

Reply 2 years ago

Great! Yes, that is correct - you have to be focused on the VRTD Sender tab for the servos to move. In an ideal situation, VRTD Sender and VRTD Receiver are on different computers, allowing you to see both screens at the same time while controlling the servos.

0
None
815669856

3 years ago

the VRTD receiver could control the servo through i update the xRot,yRot,zRot.but the VRTD receiver couldn't receive the x,y,z from VRTD sender.

6 replies
0
None
John Choi815669856

Reply 3 years ago

I am assuming you are using one computer to test both the VRTD receiver and sender simultaneously (this is fine). Try checking which window you have focused. You should have VRTD sender as the window up front and VRTD receiver in the background, otherwise VRTD sender will not register the mouse position.

0
None
815669856John Choi

Reply 3 years ago

At the present, I could control the servo with mouse moving.

But, the driver of two different camera are the same when i bought them. Then I I amend one of them to other driver .The VRTD receive only can recognize one of them.

should i update the other different driver to camera or other setting?

Thanks for helping me to debug the device.

0
None
John Choi815669856

Reply 3 years ago

Yes, you need two different webcams for this to work, as Windows cannot differentiate between two of the same webcam.

0
None
815669856John Choi

Reply 3 years ago

At the present, I could control the servo with mouse moving.

But, the driver of two different camera are the same when i bought them. Then I I amend one of them to other driver .The VRTD receive only can recognize one of them.

should i update the other different driver to camera or other setting?

Thanks for helping me to debug the device.

0
None
815669856815669856

Reply 3 years ago

now, I connect phone to computer through wifi.

0
None
John Choi815669856

Reply 3 years ago

I have another possible fix in another reply.
(Also, if possible, please comment on the Instructables page so that everyone can see potential fixes.)

0
None
Luca Steinmetz

3 years ago

I have a problem, when i run VRTD_sender or reciver, comes out an error pop up: C:\ProgramData\Oracle\Java\Javapath\javaw.exe

2 replies
0
None
John ChoiLuca Steinmetz

Reply 3 years ago

Hm, that's odd. What version of Windows and Processing are you using?

0
None
Luca SteinmetzJohn Choi

Reply 3 years ago

I solved that problem. I creat my own processing program and now it' s working.

but I have another question: how can i find the correct position of the two webcam?