loading

Secure and reliable real time data streaming is essential for IoT. I've seen plenty of demonstrations involving applications or "push button here, LED on over there" type hardware, but a friend and I wanted to make something that was more interactive... a way to almost feel the data stream as you manipulate it. So, we decided to build a motion controlled "thing" that mimics your hand movements and displays colors based on finger positioning. Hopefully, with my help you'll be able to build your own!

A conceptual overview and some more in depth look at some of the source code can be found on the PubNub blog, the complete source code is available in this GitHub Repo, and a separate Instructable detailing the LED matrix driver circuitry has been written.

Step 1: Parts and Tools

There are a lot of pieces to this puzzle, and I will try to list out the major things.

Parts and Pieces

  • Raspberry Pi - I am using a V1 Model B+
  • Leap Motion Controller
  • 5V Power Supply - There are a lot of power hungry parts here, better make it 5A!
  • I2C Enabled PWM Driver - Adafruit has a nice one.
  • 4 x Micro Servos - Cheap TowerPros will work.
  • 4 x Micro Servo Mounts - I modified these from Lynxmotion
  • LEDs and Current Limiting Resistors - For visual effect, but not necessary.
  • Project Enclosure - This one is laser cut by Ponoko, more on that later.
  • Screws, bolts, nuts, zip ties - and other such means of fastening things together
  • Cable Wrap - To keep the wires neat and tidy (on amzaon).
  • Power Jack, Switch, and Cable - So as to get power from the wall to the circuitry (on amazon)!
  • On-Off-On Slide Switch - Used to select Servo output modes
  • Small Push Button - Used as an R Pi reboot / shutdown button
  • RGB LED + 2 x 470 ohm resistors - Reboot / shutdown indication
  • Stand-Offs - To Mount componentes, These cheap ones on amazon are great.
  • Hook Up Wires - So as to connect one thing to another

The RGB Matrices sitting on top of the servos are using a custom built driver. This part of the project is fairly advanced, and a separate Instrucable has been written for it.

Software and Services

  • PubNub - Data Streaming Service. Free Sandbox mode for developers!
  • Java SE - JDK 8 - Be sure to get the correct version for your operating system
  • Leap Motion Visualizer and Java SDK - Again Be sure to get the correct package for your operating system
  • PubNub Java SE SDK - This needs to go into the project libs directory
  • Java IDE - Use your favorite, such as JGrasp, NetBeans, IntelliJ, or Eclipse
  • Project Source Code Repository - Hosted on GitHub

Raspberry Pi Setup

Tools and Such

  • Computer - Writing code, viewing of the Instructables, etc
  • Electric Drill + Drill Bits/Drivers - Drilling Holes, Driving screws, etc.
  • Dremel Rotary Tool - For cutting holes in the enclosure... may not be necessary for you.
  • Hot Glue Gun - Is there any other way to stick something to something else?
  • Soldering Iron + Solder - Don't breath the fumes!
  • Laser Cutter - Again, I used a service, but you can DIY if you have your own machine laying around.
  • Screwdrivers, Pliers - Stuff you should have by now!

Step 2: Project Overview

Like I said, there are lot of pieces to this puzzle! At the top level, there are three main components:

  1. Computer with Leap Motion Controller - publishing data to the internet
  2. Raspberry Pi in "the box" - Subscribing to data from the internet
  3. PubNub - the Communication layer securely connecting these things.

The computer part is simple enough, but in "the box" there is a bit more going on...

  1. Raspberry Pi - Primary Controller, talks to other parts using I2C Bus
  2. ATmega328p Matrix Driver Circuit - Receives commands via I2C Bus, controls 2 8x8 RGB matrices
  3. TLC5916 Based LED Sink Circuit - Controlled by ATmega328p, Sinks LED matrix current
  4. Adafruit PWM Driver - Receives commands via I2C bus, drives 4 servos
  5. 5V, 5A power supply - Brings everything to life

To start, we need to set up the user computer to use the Leap Motion Controller.

Step 3: Java and Leap Motion Controller Setup

Java Setup
We will be using the Java SDK for the Leap Motion controller, so you will need to install the Java Development Kit which includes the Java Runtime Environment. This is pretty straight forward, just choose the correct package for your operating system.

Rather than create an installation package, you will be running the source code directly from a Java IDE. This allows you to play with the code yourself, modify things, and learn! Once you have the JDK installed, you need a good IDE (that is, integrated development environment). Some common ones are Eclipse, IntelliJ, and NetBeans. Use your favorite, or pick one at random. Setting up libraries is a bit different in each, so you will have to refer to the documentation on how to setup a new project!

Java Source Code

There is only one java file needed to run the code, but you will also have to install the Leap Motion and PubNub Java libraries... The GitHub project repository contains the necessary java file. I'll talk a bit about the code in the next step.

Leap Setup

Setting up the Leap can be a bit tricky, and the process is different for different systems. Rather than step you through how I did it, I will direct you to the appropriate Leap Motion articles. You can always ask for help if you get stuck somewhere.

  1. Download the SDK, Drivers, and Applications from Leap Motion.
  2. Install the Leap Application, which also installs any necessary drivers and processes.
  3. Run the Leap Motion Diagnostic Visualizer to make sure Leap is working.
  4. Read through the Java SDK Documentation, particularly setting up a project.

PubNub Setup

The last step is to download the PubNub Java SDK. The documentation page talks about how to add the library to your Java Project.

Step 4: Java Source Code

If you are not interested in the java code, you can skip this step; however, you will at least need to put your personal PubNub keys into the code for it to work! The source code can be viewed on GitHub.

The following imports are required. If these result in any errors, you probably don't have the SDK libraries installed correctly.

import java.io.IOException;
import java.lang.Math;
import com.leapmotion.leap.*;
import com.pubnub.api.*;
import org.json.*;

It is crucial that you make your project implement Runnable so that we can have all Leap activity operate in its own thread. We begin by setting up the project main, an implementation of the Runnable interface and initializing of global variables we will be using later. The main things to take note of are the global variable "CHANNEL" as well as a pair of keys declared in main.

public class LeapToServo implements Runnable{
	public static final String CHANNEL = "leap2pi";

	// ... A Bunch of Code ...public static void main(String[] args) {
		String pubKey = "Your_Publish_Key";
		String subKey = "Your_Subscribe_Key";

		// ... A bit more code ...
	}
}

These strings are the only values that you will have to alter. The channel name can remain the default "leap2pi" or can be some other alpha-numeric value. Everything using PubNub for communication talks over a channel, so you want to use a unique name to prevent cross talk between various projects! The publish and subscribe keys are unique personal identifiers given to you by PubNub when you register for an account. Keep them safe. Keep them private. These keys prevent other people from talking on your channel(s). Take note, there might be a few places in the code that specifically use the string "leap2pi" instead of the constant "CHANNEL," and all of these instances should be changed.

The Leap Motion captures about 300 frames each second. Within each frame, we have access to tons of information about our hands, such as the number of fingers extended, pitch, yaw, and hand gestures. The values returned by the Leap for pitch and yaw (wrist pan and tilt) are in radians, which doesn't help us too much. We want to convert these values to a format accepted by the PWM driver in the box. We first convert radians to degrees, and then degrees to an acceptable servo PWM value between 150 and 600. This range corresponds to the typical working range of a servo (500 - 2000 us) as represented by the 12 bit servo driver.

The basic equation is 2500us / (2^13 -1) * (Servo Min/Max Pulse Width), so [500, 2000]us maps to [150, 600]... roughly... In the code, these conversions are handled by the following methods.

public static int radiansToAdjustedDegrees(int radians){}
public static double pitchDegreeToPWM(double degree){}
public static double yawDegreeToPWM(double degree){}
public static int normalizeDegree(int value){}

Also, please excuse the incorrect servo and PWM terminology in the code... not everyone who worked on this project was an expert in these subjects!

The best way to ensure your code is working is to load the PubNub Debug Console. From here, you can enter the name of your channel as well as your publish and subscribe keys. Hit "Subscribe," and your published data should show up in the messages box as you move your hands over the Leap Motion Controller with the Java code running.

Step 5: Raspberry Pi Setup

With the Java code working, it's time to setup the Raspberry Pi to subscribe to that data and use it to drive the LEDs and servos. You must have already configured a Raspberry Pi with a working internet connection; this can be WiFi or Ethernet, but it's up to you to get that part done! If you need help, PubNub wrote a great article a while ago. Also, you can do any of these steps directly on the Pi using a monitor and keyboard, or remotely using SSH. Either way is fine, but the end goal of the project is to have a stand-alone, headless, setup that automatically runs the necessary files on boot.

The first step is installing the PubNub Python SDK.

Open a terminal, and install the following:

  1. Python: pi@raspberrypi ~$ sudo apt-get install python-dev
  2. pip: pi@raspberrypi ~$ sudo apt-get install python-pip
  3. PubNub: pi@raspberrypi ~$ sudo pip install pubnub

Well, that was simple enough! Now we need to get a copy of all the files found in the Pi directory of the GitHub repository. The easiest way to do this is to clone the repo using git, and then get rid of all of the other stuff we don't need:

  1. > sudo apt-get install git
  2. > git clone https://github.com/pubnub/LeapMotionServoBots.git
  3. > cp -ar LeapMotionServoBots/Pi leap2pi
  4. > rm -rf LeapMotionServoBots

Now, the contents of the leap2pi directory should be identical to the files found in the Pi directory of the GitHub repo. Finally, we want the python scripts to run at boot. There are two python scripts we want to run:

  • servo.py - The code which subscribes to PubNub and drives the LEDs and servos
  • shutown.py - A shutdown button monitor to turn the Pi off.

This is simple enough, we just need to edit a Linux system file as the root user. First, make sure the scripts are executable, then open the rc.local file for editing.

  1. > chmod +x leap2pi/servo.py
  2. > chmod +x leap2pi/shutdown.py
  3. > sudo vi /etc/rc.local

You can use whatever editor you like, but I prefer vi. The last line of the file should be "exit 0" which allows for the clean exit of the file and the initialization of the Bash terminal. We can add any number of commands here, as long as they also run the final "exit" command.

Insert the following, just above the "exit 0" command:

python leap2pi/servo.py &&
python leap2pi/shutdown.py &&

Save the file, exit, and on boot, the Pi will automatically run those scripts. The "&&" parts make sure that every command is run. If you leave that off, the bash terminal will never load, so you will be locked out of your Pi indefinitely!

Step 6: Pi to Servo Driver Connections

The Pi can be used to directly drive servos, but this isn't easy. There is one dedicated PWM channel, but we need 4. It can be faked in software, but again, this isn't the easiest thing to do. Being a computer, the Pi is much better at higher level processing than low-level control. You can certainly use any PWM driver for this step, but I am using this 16 channel, 12bit driver form Adafruit. Yes, it is overkill, and yes, it is easy enough to program an ATtiny24 to command 4 servos while communicating on the I2C bus. I encourage you to create your own, but that is beyond the scope of this guide!

There are plenty of online tutorials to get you started with this board, and you will need a few additional source code files. These can be found in the Pi directory of the source code repository - the files named "Adafruit ..." will tell the Pi how to talk to the PWM driver.

Connecting the Pi to the board is easy. The Pi's I2C lines are found on Pins 3 and 5, but you also need to enable the I2C communication channel if you have not done so already. One important input pin is labeled "OE." This is an active-low "output enable" pin. The servos will only be enabled when this pin is held low. It can be connected directly to ground, or driven from another of the Pi's IO pins. In this project, I am actually driving it from the ATmega328p matrix driver circuit which also connects to the R Pi using the I2C data lines.

In the diagram, a servo is attached to channel 1 of the PWM driver. In the project, set up the servos like so:

  • Channel 0 is Left Yaw (Pan)
  • Channel 1 is Left Pitch (Tilt)
  • Channel 2 is Right Yaw (Pan)
  • Channel 3 is Right Pitch (Tilt)

Step 7: A Closer Look at Servo.py

The python code to control the servos is pretty straight forward, as long as you understand Python. Personally, I'm not so much a fan of this language, but that is beside the point. You can read along with this copy of the servo.py file on GitHub.

Just like in the Java code, you need to put your personal PubNub publish and subscribe keys in here, as well as the name of your communication channel.

The Pi will do the following on boot:

  1. Reset the AVR matrix driver circuit.
  2. Initialize PubNub with your keys
  3. Subscribe to the PubNub channel "leap2pi"
  4. Loop forever, checking an output mode switch (more on this in a minute)

Underneath the hood, the PubNub library is handling all of the hard work. This one subscribe call takes care of everything, we just need to specify a few callbacks - these are functions that are called when an event takes place.

#Subscribe to subchannel, set callback function to _callback and set error fucntion to _error pubnub.subscribe(channels=channel, callback=_callback, error=_error, connect=_connect, reconnect=_reconnect, disconnect=_disconnect)

Callbacks regarding the connection should be obvious (connect, reconnect, etc), but the one that does the most work is the aptly named "callback" callback. This function actually does something with the message we receive from the subscribed channel. As mentioned above, the box has a couple of output modes selected by a slide switch connected to a few IO lines.

  • MIRROR - The bots will mirror your movements; hence, your left hand inversely controls the right bot
  • DISABLED - The servos will be unresponsive
  • CLONE - The bots will clone your movements; hence your left hand directly controls the left bot

The logic for these modes as well as the I2C driving statements are handled in the _callback function.

The only other item of note in this file is the use of GPIO pin 4 as an output. This pin is driving the gate of a MOSFET which connects an array of blue LEDs to ground. This pin is enabled on a connection to PubNub, so the LEDs act as a connection indicator. This is a very important aspect of headless setups - there needs to be some indication to know we are successfully connected to the internet!

Step 8: A Closer Look at Shutdown.py

This script is a lot easier to follow than the last, but is just as important. Since the raspberry pi is a fully functional computer, it should be properly shut down. Just killing power to it can result in drive failure, data loss, and memory corruption.

To combat this, I installed a simple push button on the back of the box with an RGB LED indicator. Holding the button for at least one second will turn the LED blue. This will command the Pi to reboot when the button is released. Holding the button for an additional couple of seconds will cause the LED to turn red, signalling a full shutdown.

This functionality is handled entirely in the shutdown.py script. In a "forever loop," the following takes place:

  1. Sleep for 0.25 seconds
  2. Check for a button press (low state on the pin)
  3. Repeat forever
    1. On Button press, Sleep for 1 second
    2. Recheck the pin to see if the button is still being held in
    3. If button is still held in, we might want to reboot... If not, keep checking!
    4. Set RGB LED as blue
    5. Sleep for another 2 seconds
    6. Recheck the pin to see if the button is still being held in
    7. If button is still held in, we want to shutdown!
    8. Set RGB LED as red
    9. If not, reboot!

Step 9: Building the Box

To build the physical box, we had our friend Eric create a couple of vector files in Illustrator (see the attached files). These were sent to a laser cutting company (Ponoko) which sent us a bunch of laser cut pieces of wood and acrylic. The pieces came back great, but I had the joy of assembling them.

If you are careful, you can drill pilot holes in the acrylic (as well as the wood) and use thin screws to mount the pieces together. If you don't want to go that route, then feel free to build any sort of enclosure that you want. The standoffs from the parts list should be used to keep the circuit components firmly mounted to that base. I also cut a few additional holes in the back of the box to account for the power inlet, push button, and RGB led.

The most important thing is the mounting of the servo rigs. As they move around, the servos will cause any light weight mounting frame to fall over or jump around. It is crucial that they be firmly mounted to a sturdy surface - we chose the lid of the box!

The servos themselves were mounted to the Lynxmotion aluminum frames, but I did have to bend the tabs in a bit to get the servos to fit as they are slightly smaller than the intended servos for these mounting pieces.

Step 10: Wrapping Up

And that about does it! I know I glossed through a few parts, but I think a lot of this will have to be customized, especially building the actual box and servo mounts. Get creative, and do let me know if you have any questions!

Motion-controlled Servos with Leap Motion & Raspberry Pi from PubNub on Vimeo.

<p>i want to know how to connect raspbery with computer</p>
<p>Then go learn how...</p>
Im ask about fiscal conection is it by cable . Eathernet or any other conection
<p>The raspberry pi IS a computer. You connect a keyboard, mouse, and monitor to it. I have absolutely no idea what you are asking.</p>
<p>i want to know how to connect raspbery with computer</p>
<p>I fixed some problems!!! yeah!!!</p><p>but now I am facing I2C address problems. </p><p>My Pi is giving me and I have no idea.</p><p>Use GPIO.setwarnings(False) to disable warnings </p><p>GPIO.setup(15, GPIO,IN)</p><p>error accessing 0x40: check your I2C address</p><p>What does this mean??</p><p>connecting to pubnub is ok. </p><p>so I think it is something wrong with my I2C.</p><p>I already enabled the I2C...</p><p>Would you give me a tip please?( at least where I can do some research?)</p><p>thank you</p>
<p>For I2C to work correctly, there needs to be pullup resistors on both the SDA and SCL lines.. about 4.7k ohms on each should work well. I assume you are using I2C to communicate with the servo driver circuit from Adafruit...? Make sure it is powered and connect correctly, and none of the address traces have been scraped off of the PCB - that is how you set the address of the circuit (each device on the I2C bus will have its own address). </p>
<p>thank you so much for helping me. I face problems after I fix problems. on going questions. Would you help me one more time please?</p><p>so I am currently not planning to build the LED parts and the switch part. </p><p>should I change my java and rc.local file according to that?</p><p>or should I just add a switch to turn on/off the raspberry pi?</p><p>my micromotor is not even turning on.....</p>
<p>Yes. If you are not doing the LED parts, you should remove those from the Java and the Python (that sends out the commands on the I2C bus).<br><br>The rc.local file is running two python scripts. If you are not doing the shutdown button (which I do recommend, as killing power to the Pi can damage it... just like any computer) then there is no need to have rc.local run that script.</p>
<p>Hi Kurt! thank you very much for all your help. </p><p>I am so sorry to bother you but I cannot figure out how to connect links and libraries with Pubnub library files and jar files and all with NetBeans. I tried Eclipse but it is way more complicated and I think I can do with NetBeans but still I have tons of errors. I set up right with Leap Java project. but I do not know I did right for the pubnub. I registered and have keys and necessary files. but I do not know... at this moment, I just have everything in &quot;lib&quot; folder. but it says it cannot find com.pubnub file and other imported commend file. </p>
<p>Well I have been happy to help, but I'm afraid that's where it must end. If you do not understand how to link Libraries and Java Jar files, then I suggest you do a bit of research on the topic yourself. <br><br>To test if you have PubNub working, you should check out their developer debug console. Enter your keys and channel name, and you should see data packets coming through on the channel when you publish something from your java code using the leap motion.<br><br>https://www.pubnub.com/console/</p>
<p>Thank you so much. :) I did a lot of research and will see if I can solve this problem or not. I am not sure what would be a problem. my raspberry pi? motor? or programming? I will let you know when I figure this out!</p>
<p>please help. I cannot edit rc.local file.......</p><p>it says I cannot edit the already open file</p>
<p>That's an interesting message.<br><br>Are you using Sudo to edit this file? (it has special permissions).<br>Are you opening this file as read only?<br>Are you sure you have not already tried to open the file in another terminal window?<br><br>The best bet when you can't figure something like this out is to just reboot and try again. Anything that is open will be closed. Just be careful... screwing up the rc.local file can brick your system!</p>
<p>thank you so much for replying. I typed this in the terminal.</p><ol><br><li>&gt; <em>sudo vi /etc/rc.local</em></ol><p>and the file showed up on the screen. and I could not edit </p><p>so I tried to find the file under etc folder and opened it with texteditor.</p><p>and added two lines and tried to save the file. but it did not let me. I am so confused. should I download vi editor software? I thought that it was already included in the OS. would you help me? I did not open the file as read only either. </p>
<p>Did you remember to close the file (that you opened in Vi) before attempting to edit it directly? If not, that explains the error you got.<br><br>Yes, Vi(m) is included in every standard distribution of Unix bases systems, including Linux, OS X, and Raspbian (which is the R Pi version of Debian Linux). If you didn't have it, then the command would return in error when you try to open a file using Vi.<br><br>Do you understand how to use Vi? You can't just point, click, and type. Vi is a keyboard based text editor involving a lot of shortcut keys. It is extremely powerful, but has a steep learning curve. You should first learn the basics of Vi and try again!</p>
<p>I fixed the problem!!!! YEAH!!! thank you so much. yes, Vi is little advanced than my level. </p>
<p>Congrats, glad it is working for you!</p>
<p>I also have a question for the 5V power source for the PWM servo driver. I cannot find any 5V batteries. so should I use USB power source? If I need to use USB power source (additional to the power from the Raspberry pi) then how should I hook them up???? would you please help me?</p>
<p>That's not surprising. 5V is an extremely common voltage for electronics, but nearly impossible to create using raw battery cells. 3 x alkaline batteries produce 4.5V, and 4 x rechargeable cells produce 4.8V. In any case, it would take a very powerful battery to power the entire box.<br><br>The power supply I chose has a rating of 5A, and I believe this box actually uses a bit more than 4A at any given time. That means, a battery capable of sourcing about 5A of current is needed, and it needs to have the capacity to be able to do that for as long as you want the box to run.</p><p>The power for the servos and LEDs absolutely cannot be sourced from the Pi itself. That board is not capable of sourcing that much current from the pins or USB ports.</p><p>For those reasons, I highly suggest you use an actual power supply. A common USB port cannot source enough current to power the box. The only way it could is if each object were individually powered from separate USB ports, each capable of putting out an amp or two (the ones on a PC can only source 1A, but many USB chargers can supply 2).<br><br>If you have to power things individually....<br><br>The raspberry pi should have a dedicated 2A supply. It doesn't always pull that much, but it can if need be.<br><br>The 4 servos can also pull upwards of 2A (Or more) combined if they are doing much work.<br><br>The LED matrices and driver circuit require only a few hundred mA in total.<br><br>Any other LEDs or parts should have stated requirements as well.</p>
<p>thank you so much for answering my question. I used USB as an additional power source. I did not short the circuit either. YEAH! according to my research USB has 4 lines, +, ground, D+, D-. I connected + to + and Ground to -. hope I did right. </p>
<p>Yes (+) should be 5V (typically red) and (-) should be ground (typically black). You can use USB as a power source, but without handshaking between a device and USB root (called enumeration) you can never get the full allowed current from a computer port. You CAN use external USB hub or charger as 5V power sources without enumeration, however.</p>
<p>thank you so much for answering my question. I used USB as an additional power source. I did not short the circuit either. YEAH! according to my research USB has 4 lines, +, ground, D+, D-. I connected + to + and Ground to -. hope I did right. </p>
<p>Kurt, this is really outstanding! We've been trying to figure out how to move a small 12V 1amp motor for a parabolic solar hot water heater. Is there anything else we could use instead of the possible overkill Adafruit PWM? It also increases our costs considerably... </p><p>( http://www.appropedia.org/Parabolic_Solar_Hot_Water_Heater )</p><p>Again, wow!</p>
<p>Thank you. Do you need bidirectional motor control - (does the motor turn in both directions?) The adafruit PWM driver is really suited for LED or servo drivers, although you could use it for PWM speed control for normal motors. It is overkill, even for this project. I just didn't have time to make a small servo driver.</p><p>There are plenty of inexpensive motor drivers that can easily handle 12V 1A motors. They will typically require 2 control lines, and maybe one more PWM line for speed control.</p>
<p>Thnaks, Kurt! Yes, of course we need it for bi-directional control. We'll keep your suggestions in mind and again, thanks for replying!</p>
<p>all I can say is wow ! well I can also say, really,really cool </p>
<p>I just came back and voted ,, remembered I for got to the other day , good luck </p>
<p>Ha, thank you so much!</p>
<p>Wow! This is sooo awesome!! Very nicely made. Voted for you!</p>
<p>Thank you!</p>

About This Instructable

11,523views

154favorites

License:

Bio: Jack of All Trades, Master of One: Being Me!
More by Kurt E. Clothier:RV Awning Tension Adjustment Simple Steps to Give Your Robot Personality United States Photo Map 
Add instructable to: