Introduction: Hand Gesture Control: Movement of the Physical and Digital

Hey all,

So I didn't know how to properly title this but basically, in this Instructable, I am going to show you how to move a pan and tilt servo with hand movements or gestures. Also, I am going to show you how attaching a BNO055 orientation sensor to the servos will also move the digital component of this Instructable, which is optional. (If this sounds confusing, just look at the picture to get a sense of what is happening.)

Things you will need (I bought and received these as gifts from the Adafruit Industries:

- Access to Processing and Arduino IDE's

- Two Arduino UNO boards OR

- Two Breadboards

- Jumpers

- Pan and Tilt Servos

- LEAPMotion Device or

Step 1: Set Up Servos

Once you have all the materials neccessary, download the LEAPMotion SDK and IDE's for Processing and Arduino. Using the jumpers, wire the Arduinos correctly. I referenced this Fritzing schematic. One is for the pan servo and the other is for the tilt servo, doesn't matter which one as long as they are wired. Depending on which digital pin out (2-13) receives the yellow servo jumper should be noted in the Standard Firmata Arduino Sketch (in the picture, it's 2, 3. Also, be sure that it matches in the IDE/Sketches).

Step 2: Set Up Orientation Device (optional If Digital Component Is Included in Your Project)

Thanks to Kevin Townsend for the Adafruit wiring picture.

I am going to refer to the Adafruit Adafruit BNO055 Absolute Orientation Sensor Learn page for this part of the project. It's by Kevin Townsend. Here is the Link:

I attached the sensor to the servos using male-to-female jumpers which is how the sensor is extending from the breadboard. You might do the same depending on your own goals or outcomes of the project.

Step 3: Prepare the Arduino IDE's

1) The very first IDE is called Standard Firmata is responsible for connecting the data from a micro-controller (Arduino) to host computer (your PC or mac). This communication is necessary for this sketch because data from the LEAPMotion Processing Sketch will translating that data from the LEAPMotion to the pan-and-tilt servos, hopefully moving the two to the motion of your hand. Run this .ino file first.

The "Standard Firmata" sketch can be found in the "Firmata" examples drop down menu of the IDE. Make sure the pin outs on the Arduino board are the ones that are in the IDE (it's 2 and 3 now, but you can change that depending on your goals).

One thing to note is that you must make sure that the serial port is correctly synced with the servo Arduino. You can find port in the "tools" drop down menu of the IDE (when the Arduinos are plugged, they should be lablelled as COM1 or COM 2 or something like that. If you are having trouble with this, go to your device manager, find the Arduinos, and right-click on them and change the properties so the proper COM serial port is showing. If there is any other troubles refer to this guide:

2) The second Arduino IDE can be found here Be sure that you extract the files into the proper place of your PC, so that when it is called in the IDE, there are no errors when it is running. Run this .ino file next.

Thank you Tony Dicola.

Again, this is only necessary if you want a digital aspect of this tutorial.

Step 4: Set Up Processing Sketches (the Visual Aspect)

So here comes the visual aspect of this tutorial.

Once you have everything wired and connected, run the arduino sketches.

Run Standard Firmata.ino and Bunny.ino file and make sure no errors pop up (make sure the correct serial port are established).

With processing, make sure you download it and have these libraries installed.

Arduino (Should be in the Processing library downloads folder).

LeapMotion P5 (for reference of fingers and hand:

for digital component (optional)....

saito obj loader (also, have a 3D model. Should be in the library folder of Processing).

1) Here is the LEAPMotion sketch that I have been using with minor edits(such as the com serial port and 3D model for instance). This sketch runs with the Standard Firmata Arduino IDE from the previous step. Again, make sure that files are in the correct assets directory so that when you call it in the sketch, everything runs smoothly. This is for the LEAPMotion gesture control that moves the pan and tilt servos. Run this sketch.

2) Here is the sketch that takes care of the sensor information which dictates the orientation of the 3D model. This sketch runs when the bunny.ino file is already running from the previous step. Finally, run this sketch. Make sure that this amtches the COM port attached to the bunny.ino from the previous step. This will translate the data from your hand that the LEAPMotion is recording into the servos and the sensor/digital aspect of this tutorial as demonstrated in the GIFS.

Step 5: Things to Remember

(Video starts at :29)

Just make sure the COM Ports in the Arduino IDE's and the device manger are defined and remember which Arduino pairs with which Processing sketch. All in all, there is only 2 Processing sketches and 2 Arduino IDE's. Make sure your assets are in their proper settings so calling them should not be difficult.

I had fun making this and there were some struggles, obviously, this is a rather meticulous process but it functions and works. Just like all of you reading this, I am still learning.

Feel free to ask questions and I will do my best to answer them. Also, post your successes and/or critiques/criticisms.

Also, if this is difficult to read and there are errors or anything like that, lemme know.

(If there is any problems with Processing or Arduino, consider downloading previous versions. I had success with Processing 2 but I believe any Arduino version should work.