Instructables

Kinect controls Arduino wired Servos using Visual Basic 2010

Picture of Kinect controls Arduino wired Servos using Visual Basic 2010
Kinect2Arduino (1).jpg
Kinect2Arduino (2).jpg
Kinect2Arduino (3).jpg
Kinect2Arduino (4).jpg
Kinect2Arduino (5).jpg
Kinect2Arduino (6).jpg
Kinect2Arduino (7).jpg
Kinect2Arduino (8).jpg
Kinect2Arduino (9).jpg
UPDATE JAN 2012: This guide is featured on Microsoft's MSDN Channel9
http://channel9.msdn.com/coding4fun/blog/Kinecting-to-Arduino-with-Visual-Basic

INTRODUCTION

In this project, I use left and right hand movements in Kinect to control the rotation of two servos tied to Arduino Uno, which in turn moves the left and right arms of a makeshift stickman. The middleware is written in Visual Basic 2010 and is based on the Microsoft Kinect SDK beta 2.

There are plenty of C++ and C# code examples to interface Kinect to other gadgets, but not many Visual Basic examples. So I figured this project could help folks with mostly VB skills to start hacking Kinect with Arduino.




HOW THIS STUFF WORKS

Kinect keeps track of 20 skeleton joints for each player.  To simplify this project, I extract the coordinates of the left hand and right hand joints only. Then I pass those joint coordinates to an Arduino UNO after I do some scaling in Visual Basic from computer screen size to servo angles. 

I modified the VB code in the Microsoft Quickstart samples because it's well documented and comes with good tutorials. So anyone can review these tutorials to learn more about how this VB code works with Kinect.

The changes I made to the sample VB code were mostly to facilitate serial communications with Arduino as well as to map screen coordinates to the servos angles. In this project, I capture the Y (vertical) movement of the HandLeft and HandRight joints. It should be easy to modify my code to add more servos and joints.

SETUP

Install all the required Microsoft software following the instructions provided by the vendors. Refer to Microsoft's Kinect for Windows SDK Quickstarts http://channel9.msdn.com/Series/KinectSDKQuickstarts Microsoft did a great job with these tutorials.

The biggest challenge is getting the various packages to work together.  When you successfully run the Kinect demo provided by The Microsoft Kinect SDK, you are half way there. 

The second part is wiring the servos to the Arduino and running my code. 

I kept my code as simple as possible so there's little error trapping and no fancy user interface. You can add that per your requirements.  


WIRING

For this project, I used two TG9e micro servos powered by the Arduino Uno connected to my PC via USB. For bigger servos with more power needs, you may have to power them separately or else the Arduino will act erratically.

I labeled one servo as the Left Hand servo while the other as the Right Hand servo. If you confuse the servos at some point, the fix is often as simple as changing a line of code instead of rewiring.

Left Servo --->  Arduino Digital Pin 9
Right Servo --->  Arduino Digital Pin 11 

For debugging purposes, you may wish to consider testing your build with one servo before adding another.

HOW TO APPLY MY CODE

In addition to the instructions, you can view my step-by-step screen captured stills. Please click the "i" in the upper left corner of each photo to see the full resolution still.

- Download the "ArduinoXYZ.ino" file (part of the attached techbitar.zip file) and load it into your Arduino UNO via the Arduino IDE 1.0.

- Download KinectForWindowsSDKQuickstarts.zip

-  Find and extract the SkeletalTrackingVB folder and place it under the Projects folder for your Visual Basic 2010 Express.

The SkeletalTrackingVB project files you have extracted have old reference files from an older Kinect SDK and Code4Fun toolkit. We need to delete the old references then add the latest ones.

NOTE: When following my videos or screen captured instructions, please keep in mind that I may have extracted the project and support files to different directories than yours.

- From Visual Basic 2010 IDE explorer, double click on MyProject. You will be presented with a project configuration page.

- Click on the References tab.

- Search for Microsoft.Research.Kinect and delete it.

- Search for Code4Fun.Kinect.wpf and delete it.

- Click on Add.

- Click the Browse tab.

- Search for and add the newer versions of those files which you have downloaded part of the latest Microsoft Kinect SDK and Coding4Fun Kinect Toolkit zip files. As of the writing of this guide, the versions for Code4Fun.Kinect.wpf should be 1.1.0.0 and for Microsoft.Research.Kinect 1.0.0.45

- Download my MainWindow.xaml and MainWindow.xaml.vb (they can be found in the attached techbitar.zip file) which I have attached to this guide and copy them on top of the ones you have extracted under SkeletalTrackingVB\SkeletalTracking folder or cut and paste their content over the extracted files with the same names.

With the Kinect and Arduino connected, you should be able to test the program.

NOTE: The COM port field on the main application window is set to 16. Change this to match the Arduino's COM port on your PC.

KINECT POWER SUPPLY

The Kinect I am using in this project has its own power supply attached to the USB cable.  I bough this as a stand-alone without the XBOX. Some Kinects don't have an external power supply and get their power from the XBOX. If you have one of those, you will need to buy a Microsoft Kinect power supply which seems overpriced. So before you buy a Kinect/XBOX bundle, check if the bundled Kinect can be powered and operated as a stand-alone.


HARDWARE PARTS

- Microsoft Kinect
- Arduino Uno
- Micro Servos X 2 (I used TG9e micro servos)
- Breadboards and wires

SOFTWARE COMPONENTS

Arduino IDE 1.0
http://www.arduino.cc/en/Main/software

Microsoft Kinect SDK V1 beta2
http://www.microsoft.com/en-us/kinectforwindows/download

Coding4Fun Kinect Toolkit 1.1
http://c4fkinect.codeplex.com/releases/view/76271#DownloadId=300233

Microsoft Visual Basic Express 2010
http://www.microsoft.com/visualstudio/en-us/products/2010-editions/visual-basic-express

.NET Framework 4
http://msdn.microsoft.com/en-us/netframework/aa569263

Microsoft DirectX SDK - June 2010 or later version
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=6812

Runtime for Microsoft DirectX 9
http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=35

Microsoft Kinect Code4Fun Quickstart VB code (You will overwrite this with my modified version)
http://files.ch9.ms/coding4fun/KinectForWindowsSDKQuickstarts.zip


REFERENCES

Kinect for Windows SDK Quickstarts
http://channel9.msdn.com/Series/KinectSDKQuickstarts

Open Kinect
http://openkinect.org/wiki/Main_Page

Kinect Hacks
http://www.kinecthacks.com/

Kinect Motion Capture
http://www.brekel.com


CONTACT
Developer: Hazim Bitar
Email: techbitar {at} gmail {dot} com
isaidkuchy1 month ago

Hi I am engineering student. I would like to deploy this project could provide me the visual code?

bdaldegan6 months ago

Olá, bom dia!

Eu não posso fazer o download do link:
http://files.ch9.ms/coding4fun/KinectForWindowsSDKQuickstarts.zip

É possível me enviar por e-mail?

brunodaldeganlimao@gmail.com

Obrigado!

grande trabalho!

Bob Bowie9 months ago

Hopefully, you can send me the Visual Basic code and show me how to let my PC control this remote controlled lawnmower:

http://www.instructables.com/id/Remote-Controlled-Lawnmower/

The above web site has a video of my remote controlled lawnmower.

Currently, the wheel chair motors are controlled by a Futaba 4YF transmitter. I want to eliminate the Futaba 4YF transmitter. To do this, I want my Windows 7 PC using Visual Basic to control the wheel chair motors with an interface to an Arduino board. The Arduino will send commands to the 2 servos that control each of the 2 wheel chair motors. The Visual Basic program interface needs to allow the following 2 steps:

Step #1-Learn Mode-Control the mower via the Arduino interface to the wheel chair motors and have the Visual Basic program record the sequence of commands sent to the Arduino.

Step #2-Playback Mode-Transmit commands recorded on the PC in Step #1 to the mower.

What is the easiest way to accomplish these 2 steps?

Csl2 Bob Bowie8 months ago
Did u ge the visual basic code?

Can you give it to me? The link doesn't work.
Bob Bowie Csl28 months ago
Sorry, but the approach/steps you outline are way over my capability.

I'm well versed with the old DOS Basica, but am a novice when it comes to Visual Basic. The remote controlled lawnmower works really well and could have commercial appeal if the 2 servos could be controlled by a PC. Any interest in working with me to let a PC control the lawnmower as previously outlined?
You2139 months ago

this link is broken): Microsoft Kinect Code4Fun Quickstart VB code (You will overwrite this with my modified version)

http://files.ch9.ms/coding4fun/KinectForWindowsSDKQuickstarts.zip

josephcxhxa9 months ago

will you be so kind if you reupload Microsoft Kinect Code4Fun Quickstart VB code please.? the link is no longer online

Microsoft Kinect Code4Fun Quickstart VB code
kevinomaralmendarez@gmail.com
Really thought this project was awesome. I have actually modified the code so that I can operate 4 servos at one time configured as an arm with gripper attachment. I noticed in a comment you posted below about the joint numbers ranging from 1-20 if possible I was wondering if you could list the all the joints that are possible along with the corresponding numbers as I would like to see how my project responds to alternate joint positioning. Again love your work.
mr921 year ago
Hi, great tutorial but i am having trouble. For some reason the VB does not communicate with the arduino. I only have one com port which is com3 and i have changed it in the coding.
Malrysson1 year ago
Ola gostaria de saber qual o nome da linguagem que foi utilizado no arduino??
e se teria como explicar melhor sobre os códigos tanto no VB como no Arduino??
SkiRa3rDud32 years ago
Techbitar: Would you be able to update the code so it can be used with the Kinect for Windows SDK v1.5 ? I have been trying but the more i "fix" the more i break :P
techbitar (author)  SkiRa3rDud32 years ago
I promise to so at earliest available opportunity.
eva222 years ago
hi i've followed all the steps but finally nothing happend the servo does not move ,it seem like the arduino does not recive any thing from the kinect .... can any one help me please ?
You'll need to provide more details. What version of the Kinect SDK are you using?
SkiRa3rDud32 years ago
Quick question; in the arduino code, what does the j and z values represent?
techbitar (author)  SkiRa3rDud32 years ago
can you post the code segment in question
void loop()
{
// read the 4-byte byffer sent from VB containing coordinates and joint ID.
if (Serial.available() >= 4) {
x = Serial.read();
y = Serial.read();
z = Serial.read(); <<<< j = Serial.read(); <<<<
if (j == 7) { // HandLeft
lservo.write(y); // sets the servo position according to the scaled value
delay(10); // waits for the servo to get there
}

if (j == 11) { // HandRight
rservo.write(180-y); // sets the servo position according to the scaled value
delay(10); // waits for the servo to get there
}

}
}
techbitar (author)  SkiRa3rDud32 years ago
My VB program sends 4 pieces of info to Arduino. X,Y,Z which are the coordinates of tracked body joints relative to the position of the Kinect sensor. J is the joint number. Kinect assigns numbers from 1 to 20 to refer to each one of the joints it tracks. Joint 7 is the left hand and joint 11 is the right hand. These are the only joints I am reading for my paper puppet to move right and left servos. My VB code sends all Kinect joint info to the Arduino. You can change my Arduino code to read other joints to do more elaborate robotics tricks. Hope this helps.
Thanks, I'm planing on making your sick man with both moving arms and legs
techbitar (author)  SkiRa3rDud32 years ago
It's very doable.
sxkx3602 years ago
Great tutorial, had some mix-up with the arduino COM I/O but sorted it out changing arduino port to match the program(16).
 In an instance I also managed to modify the arduino code so that i can control one hand in x and y axis (just typing x instead of 180-y and matching the j-value to 11). THANK YOU!!
techbitar (author)  sxkx3602 years ago
Thanks for the feedback and glad it worked for you.
Techno742 years ago
I never thought hacking Kinect can be so simple. THANK YOU!