Robotic Hand Controlled by Gesture With Arduino + Leap Motion

59,472

243

63

Published

Introduction: Robotic Hand Controlled by Gesture With Arduino + Leap Motion

About: Hello! My name is Alessio Sacchetta Cali and I'm an electronic engineer specialized in biomedical devices. I'm also a web developer, I worked for so long as freelancer and now I'm the co-founder and IT mana...

Hello everyone,

As promised I have made a full step by step instruction for my university final project!

First of all: sorry for my bad english! If something isn't clear, please leave a comment.

The hand is controlled by Leap Motion + Arduino Uno (Standard Firmata, Nodejs & Johnny Five) + Servo Motors and the hand itself was made with Polymorph. It can recognize each finger and it can move them separately with very good precision and sensibility! This prototype can be controlled remotly too via a normal web server using ajax/json requests. Let me know what you think about it and maybe I will publish a complete tutorial when I will have more time... Thank you!

--- Feedback from NASA ----------------------------------------------------

“ Your hand is very responsive and does a good job of showing how well your gesture control/recognition works. ”

Ron Diftler, Ph.D. Robonaut Project Manager, NASA/Johnson Space Center -------------------------------------------------------------------------------------

Step 1: Parts

Parts: (Cost will be about 300$)

  1. x1 Arduino UNO
  2. x1 Leap Motion
  3. 0.5kg of Polymorph
  4. About 3m Fishing Line (0.17mm - Spiderwire)
  5. Some elastic strings
  6. x5 Servos (TowerPro MG956R)
  7. x1 Battery 1600mah and 6V (NiMH Venom VEN-1504)
  8. x1 Small Breadboard
  9. x1 Hot gun
  10. Some wires and tools

Step 2: Make the Hand!

  1. Put all polymorph spheres inside a basin with hot water
  2. Mix up all the spheres in a unique piece of polymorph
  3. Divide the big piece of polymorph in some smaller pieces
  4. In order to make each fingers you have to draw a scheme using your hand (see picture)
  5. Take a piece of polymorph and put it in a microwave oven for like 30s @ max power
  6. It will become transparent and easy to model. Be careful because it is very very HOT
  7. Use hand sketch to adjust the size of the finger and model the piece of polymorph, using some pilers to make all the joints. Joints are the tricky part, they should blend just in one direction!
  8. Make all five fingers and let them cold down
  9. Now you have to make the palm of the hand: put a bigger piece of polymorph in the microwave oven
  10. Model the piece in order to obtain something similar to a hand palm: before it cold down you have to put all fingers inside the warm palm. Be carefull to put all the fingers in the right position because once everything is cold, it would be impossibile to fix possible problems!
  11. Now it's time for some wire loops. We need to make some wire loops on all fingers (on the front, for the fishing line and on the back, for the elastic line) in order to avoid wire intertwine during fingers movements. On each finger you have to make a hole with a hot screwdriver, in between of all joints and on all finger tips (so 4 holes per finger, see pictures)
  12. Make some little balls of polymorph and put them in the microwave oven, press one small ball of hot polymorph inside a hole and model it using a toothpick to make a hole throught it (see picture). Fishing line and the elastic one will pass through all these wire loops!

  13. Make some wire loops on the hand palm too
  14. Finally we have to make the arm part, that will host all servo motors. Take a A4 white paper and draw all servo motors shapes (See picture).
  15. Put a big piece of polymorph in the microwave oven. When it's hot, model it on the drawn schema and use a cutter to make all holes for the servo motors.
  16. When the arm is cold you will have to put all servo motors inside the holes and seal them using some hot polymorph strips. Be carefull to put all servo motors in the right orientation (See picture), two of them will be inverted.
  17. You have to attach the arm to the hand by using a hot gun and some extra hot polymorph
  18. When the hand and the arm are fused together the last step consist of connecting all servo motors with each fingers: you can fuse an extremity of a piece of fishing line at the tip of the finger by using a common lighter whille the other extremity can be simply tied on the servo motor (see picture
  19. It's raccomended to create some wire loops to put on each servos in order to avoid wire interwine while servos will move.
  20. Finally you can put some elastic line on the back of each finger (see picture), make sure that all elastic lines are well strained: this part will give your hand the ability to keep the relaxed position even when the hand is oriented with the palm looking down!

Now you have finished the mechanical part of the project and you are ready to give life to your artificial hand :D

Step 3: Attaching All Servos to the Arduino

Now that we have our hand and all servos implanted on the arm we must use the little breadboard in order to connect all servos to the Arduino Uno and to the battery pack:

  1. Each servo come with 3 wires, the black one for the ground, the red one for the power supply and the yellow one for the PWM (important!) signal
  2. By following the schema in the picture, you should connect all servos wires in the breadboard
  3. Now it's time to use some wires to connect all grounds and all power supplies (just follow the schema)

    Attention! I didn't pay attention to use right colored wires (because I didn't have all required colors!)

  4. Take a common ground and connect it to the Arduino Uno ground input
  5. Take another common ground and connect it to ground contact of the battery pack
  6. Attention! Don't connect the common power supply to 5V input of the Arduino Uno (as showed in the schema), you have to connect this to the battery pack!
  7. Finally you have to connect all servos yellow wires to a PWM digital input on the Arduino Uno. I have used ports 3, 5, 6, 9 and 10. Attention! Try using these ports because when we will come up with the code, I will assign servos variables to those digital ports!

Now you have the hand, all servos implanted, connected to the Arduino Uno and the battery back.

Step 4: Preparing the Developing Platform and Leap Motion

Now it's time to prepare software platform in order to program the arduino board. Since Leap Motion sends its data throught a Web Socket, we will use Google Node.js as javascript platform (it's based on Google Chrome js engine) so that we can easily read data via the Web Socket. Then, we should install a Node.js module called "johnny-five" that allow us to program arduino board directly with javascript.

NOTE: I did everything with a Macbook Pro so please don't ask me how to solve problems on Windows systems because I can't really help you! Most of problems can be solved by searching on google.

  1. Plug in Leap Motion and install its software. At the end you should be able to see an icon somewhere, that allows to open the AirSpace software and Leap Motion configurations.
  2. Inside Leap Motion configuration you should set "Precision" and make the "Recalibration" tutorial. Please note: Leap Motion surface should be very clean!
  3. Now that Leap Motion is connected, installed and working we have to install Node.js so go to http://nodejs.org and click on the "Install" button (it's possible that first you need to download and install XCode in order to install nodejs, I already have it installed so I can't be sure). You can check Node.js successful installation by going on Terminal and type "node" and press "enter". If nodejs console show up starting with a ">" then you are ready to go. If you have some problems with the installation, please search on google some tutorial because there are a tons!
  4. Now you have to install some nodejs modules (using npm install syntax). Again, check on google how to install a module it's very easy btw! Needed modules are the following:
    1. gl-matrix: matrix and vector library
    2. johnny-five: arduino library
    3. underscore: javascript support library
    4. ws: websocket library
  5. Install LeapJs SDK for Node.js check this url: https://developer.leapmotion.com/leapjs/welcome

  6. I have installed everything inside a project folder called "Tesi". Inside this folder I have 3 folders:
    1. "lib" containing the "lib" folder of LeapJS: check on github https://github.com/leapmotion/leapjs
    2. "node_modules" containing all 4 modules folders
    3. "script" containing my script

Please, try to keep this project folders structure and paths because unpredictable errors may happen if not.

Step 5: Algorithm and Scripts

This is the last step and the most important one. Now that we have our working artificial hand connected to all servos controlled by Arduino Uno and the developing platform is ready-to-use... We really have to make the software in order to:

  1. Retrieve Leap Motion data
  2. Elaborate the data to assign detected objects by Leap Motion to appropriates fingers (servos)
  3. Elaborate the data to obtain appropriates servos angles
  4. Send elaborated data to Arduino Uno in order to move servo motors

Attached to this step there is my full project folder. The script that you should run in Node.js is "/script/app.js" (make sure that leap motion and arduino are connected to your computer, the arduino should be connected to the battery pack and all servos should be attached to the Arduino board too).

Just for a matter of order I have put all core functions inside another file inside "/lib/handToHand.js" that is imported at the very beginning of the "app.js".

I have rewritten all my comments in english so that anyone can understand what's happening and why. If you follow my comments and check the flowchart (see picture) you will understande everything for sure.

If you have any inquiries please leave me a comment!

That's all... Thank you and good luck!

-----------------------------------------------------------------
Alessio Sacchetta Cali
Electronic Engineer
University of Roma Tre

2 People Made This Project!

Recommendations

  • Creative Misuse Contest

    Creative Misuse Contest
  • Clocks Contest

    Clocks Contest
  • Water Contest

    Water Contest

63 Discussions

Hello,

I tried to run your script and I have an error message :

"C:\Users\psaur\Desktop\leap motion\projet_main_instructable\script>node app.js

It looks like serialport didn't compile properly. This is a common problem and its fix is well documented here https://github.com/voodootikigod/node-serialport#...

C:\Users\psaur\Desktop\leap motion\projet_main_instructable\node_modules\johnny-five\node_modules\firmata\lib\firmata.js:28

throw "Missing serialport dependency";

^

Missing serialport dependency"

Could you help me to solve this problem?

I tried to change the folder but still not running!

Thanks a lot.

Philippe

1 reply

Did you ever get this running? If not let me know some more information about your setup. i.e. OS, etc. I had originally tried this on windows 10 and it didn't work. I tried again on a mac os and it ended up working. Also you have to make sure that you have the standardfirmata code running on your arduino. Let me know if you have any questions. I'm happy to help if I can.

Has anyone tried to run this code recently? Just trying to see if anyone can verify that the code still works before trying to interface the leap motion.

1 reply

I just got it up and running two days ago and it works. You have to make sure the standardfirmata code is running on your arduino. I also took the handtohand code and pasted it into the app.js file. Please let me know if you have any questions and I'll help if I can.

0
user
acali

1 year ago

Hello everyone,

Source code was provided as well as you can see in the attached screenshot to this comment. Please consider that the code was working with previous versions of leapjs, nodejs and all the modules!

2016-10-19_17-27-34.jpg
1 reply

I have almost everything to get the project started besides the leap motion. Before buying it I just wanted to know if anyone could verify that the program still works with the updated versions of leapjs and modules.

Thanks!

Hi. Where did you store your "Tesi" folder? Mine is in My Users folder and I keep getting this error: "SyntaxError: Invalid regular expression flags"

Thanks in advance!

broken link :/ can u send me the details and code suleymankoc315@gmail.com

0
user
Ilavic

1 year ago

Amazing project , can u please share the code

plz send to ali.dmc94@gmail.com

Sir is it possible to give me the code of this project.

Please send me at menegassi.bta@gmail.com

I want to make same can u give me guide how i start making coding

Its great can u shear the code.... immadfareed@gmail.com

Sir is it possible to give me the code of this project. That will help me a lot

amazing project sir. can i get the coding for this so that i can understand and learn

emailid: jazzymohamed@gmail.com

Amazing..sir can i get the code of this.It will be very useful for my project sir...Thank you.. sir..

chathurangakila@gmail.com

Hi Alessio!

Can i use leap motion controller to send it's data to arduino wifi shield.

Actually i want to use leap motion to make a robotic hand, which can be operated from another room.Is it possible?

Thanks in advance.

sir can please share the code of the project.

mikrajiisra@yahoo.co.id

sir could u please share the code of the project?

harryalexendro1996@gmail.com

sir can please share the code of the project.

bkwatra999@gmail.com

Hello Acali,

Could you please share the coding of this mdave202@yahoo.com