How to Control Almost Anything With a 3D Camera (including Your Arduino)

If you control your application by keyboard, mouse or joystick then it’s easy to control it with hand gestures using a 3D camera. This tutorial will show you how it can be done using Device Emulator, a neat open source piece of software.

Bonus: DeviceEmulator can also control your Arduino! 

Step 1: Select a Camera and Install the Drivers and Software

1. Get a camera that is compatible with SoftKinetic’s iisu middleware:

For close interaction aka finger tracking (10cm to 1m) use the Softkinetic DS325 or the Creative Senz3D
For far interaction aka full-body tracking (1m to 4m) use the SoftKinetic DS311, Mesa Swissranger 4000, Panasonic D-Imager or Asus Xtion.
In this tutorial we use the Creative Senz3D.

2. Install the camera’s drivers and plug your camera into your computer (follow the manufacturer's instructions)
3. Download and install SoftKinetic’s iisu middleware

iisu analyzes the 3D images given by the camera and provides hand gesture or full body information.
Go to www.softkinetic.com, click on the ‘Download iisu’ button on the homepage, download iisu Free (you will need to register first) and install it.
Launch and test the application.

4. Download and unzip DeviceEmulator.zip (here attached).

(You can find the source code there: https://github.com/fwindeySK/iisuEmulator/)

5. if you are interested in emulating a joystick then download and install Vjoy Virtual Joystick Driver V1.2 (you will find atthe bottom of that page)

Now that everything is installed you can launch DeviceEmulator.exe and start emulating

Step 2: Choose What You Want to Control and How You Want to Control It

Now you can launche DeviceEmulator.exe and start emulating...

1. Load one of the scripts by clicking on the "..." icon.

The scripts are where iisu computes the values that can be linked to our controls.

• If you have a close range camera use DeviceEmulator\CloseRange\CI_Example_Script_With_Gestures.iid or any file from the CloseRange folder
• • If you have a long range camera, use DeviceEmulator\FullBody\GestureLibrary_Hands_Manipulation.iid... or any file from the FullBody folder

Once the script is loaded, you will see the values that are computed in the script. You can map them to the devices you want to emulate. 

2. Choose the device(s) you want to emulate by clicking on "none" in the "Emulator" column.
3. Depending on the device, you can specify what you want to emulate.

• For a keyboard, choose which key will be mapped to your value. Keyboard emulation only works with Digital Values (on/off).
• For a joystick or a mouse:
  • With Digital Values (On/Off), choose which button you want to map
  • With an Analog Value (continuous value), choose which axis you want to emulate
• For Arduino, you have to choose the Pin and Pin Type. More details on how to use the Device Emulator with an Arduino on the next step.

4. Optional step: Choose a value to start and stop your emulation.

This is useful when you emulate your mouse and you want to control it back as soon as no one’s body or hand is detected.  You might struggle to control your computer with anything else than gestures. 

5. Optional step:  press "Open in IID" if you want to edit the script and generate new values
6. Press "play" to start your emulation

Step 3: Control an Arduino Board

The device emulator can also control Arduino Pins!

For this we use the Firmata protocole. Its fairly simple to setup.

1. Upload the "StandardFrimata" sketch to your Arduino.You can find it in the Arduino IDE in File>Examples>Firmata>StandardFirmata
2. In the DeviceEmulator folder, edit the Settings.ini file with a text editor to set the Com Port on which the Arduino is plugged
3. Choose your values mapped to the DeviceEmulator by following the instructions from the previous step..

You are now ready to make things move with hand gestures :o))