Amazing Control Computer Using Hand Motion and Arduino

In this project,we are going to learn how to build a technique of gesture controlled Laptops or computers,It based on using the power combination of Arduino and Python.

Instead of using a keyboard, mouse or joystick, we can use our hand gestures to control certain functions of a computer like play/pause a video, move left/right in a photo slide show, scroll up/down in a web page and many more,This is why I decided to control VLC Media Player as hand gestures project.

The idea behind the project is quite easy by using two Ultrasonic Sensors (HC-SR04) with arduino,We will place the two Sensors on the top of a laptop screen and calculate the distance between the hand and the sensor.Counting on the information from Arduino that is send to the python through the serial port,This information will be then read by python which is running on the computer in order to perform certain actions.

Hardware components :

1. Arduino

2. Two Ultrasonic Sensors (HC-SR04)

3. Some wires

Software apps :

1. Arduino IDE

2. Python IDLE

3. PySerial library (We will use to communicate with serial ports).

4. PyAutogui library (We will use to perform actions).

So I assume that you have already installed Python and PySerial library and have successfully done some basic projects. If not, don’t worry I advice you to follow my previous tutorial(Programming Arduino Using Python).Also if you want to get acquainted with the Ultrasonic Sensor you can take a look (here).

Step 1: Watch the Video for More Details

Step 2: Hand Gestures Purpose

The following are the 5 commands hand gestures that I’ve programmed for demonstration purpose:

First gesture hand: It allows us to 'Play/Pause' VLC by placing the two hands in front of the Right/Left Ultrasonic Sensor at a particular far distance.

Second gesture hand: It allows us to 'Rewind' the video by placing a hand in front of the Left Sensor at a particular far distance.

Third gesture hand: It allows us to 'Forward' the video by placing a hand in front of the Right Sensor at a particular far distance.

Forth gesture hand: It allows us to 'Increase Volume' of the video by placing a hand in front of the Left Sensor at a particular far distance and moving away from the Sensor.

Fifth gesture hand: It allows us to 'Decrease Volume' of the video by placing a hand in front of the Left Sensor at a particular far distance and get near to the Sensor.

Step 3: Schematic

The schematic is quite simple you should just follow the instructions bellow,So each sensor has 4 pins :

1.pin Vcc - this pin is connected to 5V+.

2.pin. Trig - you need to define this pin in your program.

3.pin Echo -this pin is the same as Trig you also need to define him.

4.pin GND - this pin is connected to ground.

Step 4: Programming Python

1. Install Python IDLE :

You can skip this step if you have installed the Python IDLE already in your computer.If yes then go to step 2 or else look to the following instructions:

1.Go to the python website and download it (here).

2.Once you have done,you move on to installation by keeping the directory in which the python is getting installed by default.

NOTE:Even if your Computer is operating on 64-bit you can use 32-bit Python itself, due to the lack of compatibility with Arduino Libraries.

2. Install PySerial library :

PySerial is a Python API module which is used to read and write serial data to Arduino or any other Microcontroller.To install on Windows, simply visit PySerial's Download Page and following the steps bellow :
1.Download the PySerial from the link above.

2.Install it by keeping the setting as the default You should be sure that Pyserial worked correctly,To do this;You type in :

import serial

If you haven't confronted any error,so you're in the good way,if else I advice you to check your installation and Python IDLE extension.

3. Install PyAutogui library :

The purpose of PyAutoGUI is to provide a cross-platform Python module for GUI automation for human beings. The API is designed to be as simple as possible with sensible defaults.Follow the below steps to install pyautogui for windows.(To follow this steps you should be already had installed Python IDLE)

If you are using other platforms I advice you to take look (here).

1. Open Windows Command

2. Type in the following command

cd C:\Python27

3. Then you type :

python –m pip install –upgrade pip

4. The last command is :

python –m pip install pyautogui

NOTE: To be sure that pyautogui worked correctly just type in :

import pyautogui

If you haven't confronted any error,so you're in the good way.

Step 5: Arduino Code

To initiate a connection with the Arduino from Python, we first have to figure out which COM Port the Arduino is on. This task is simply made by the Ardunio programming environment as I notify in the picture above.

<p>const int trigger1 = 2; //Trigger pin of 1st Sesnor<br>const int echo1 = 3; //Echo pin of 1st Sesnor
const int trigger2 = 4; //Trigger pin of 2nd Sesnor
const int echo2 = 5;//Echo pin of 2nd Sesnor</p><p>long time_taken;
int dist,distL,distR;</p><p>void setup() {
Serial.begin(9600); 
  
pinMode(trigger1, OUTPUT); 
pinMode(echo1, INPUT); 
pinMode(trigger2, OUTPUT); 
pinMode(echo2, INPUT); 
}</p><p>/*###Function to calculate distance###*/
void calculate_distance(int trigger, int echo)
{
digitalWrite(trigger, LOW);
delayMicroseconds(2);
digitalWrite(trigger, HIGH);
delayMicroseconds(10);
digitalWrite(trigger, LOW);</p><p>time_taken = pulseIn(echo, HIGH);
dist= time_taken*0.034/2;
if (dist>60)
dist = 60;
}</p><p>void loop() { //infinite loopy
calculate_distance(trigger1,echo1);
distL =dist; //get distance of left sensor</p><p>calculate_distance(trigger2,echo2);
distR =dist; //get distance of right sensor</p><p>//Pause Modes -Hold
if ((distL >40 && distR>40) && (distL <60 && distR<60)) //Detect both hands
{Serial.println("Play/Pause"); delay (500);}</p><p>calculate_distance(trigger1,echo1);
distL =dist;</p><p>calculate_distance(trigger2,echo2);
distR =dist;</p><p>//Control Modes
//Lock Left - Control Mode
if (distL>=13 && distL<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger1,echo1);
  distL =dist;
  if (distL>=13 && distL<=17)
  {
    Serial.println("Left Locked");
    while(distL<=40)
    {
      calculate_distance(trigger1,echo1);
      distL =dist;
      if (distL<10) //Hand pushed in 
      {Serial.println ("Volume Increased"); delay (300);}
      if (distL>20) //Hand pulled out
      {Serial.println ("Volume Decreased"); delay (300);}
    }
  }
}</p><p>//Lock Right - Control Mode
if (distR>=13 && distR<=17)
{
  delay(100); //Hand Hold Time
  calculate_distance(trigger2,echo2);
  distR =dist;
  if (distR>=13 && distR<=17)
  {
    Serial.println("Right Locked");
    while(distR<=40)
    {
      calculate_distance(trigger2,echo2);
      distR =dist;
      if (distR<10) //Right hand pushed in
      {Serial.println ("Rewind"); delay (300);}
      if (distR>20) //Right hand pulled out
      {Serial.println ("Forward"); delay (300);}
  }
}
}</p><p>delay(200);
}</p>

If you read the arduino code,you will observe 5 commands which control certain Keyboard Functions in order to achieve the required task.

arduino_pythogui.ino
Download

Step 6: Python Code

First up, we need a simple program to get the Python sending data over the serial port.

import serial #Serial imported for Serial communication
import time #Required to use delay functions
import pyautogui   #Required to to perform actions
ArduinoSerial = serial.Serial('com15',9600) #Create Serial port object called arduinoSerialData
time.sleep(2) #wait for 2 seconds for the communication to get established
while 1:
    incoming = str (ArduinoSerial.readline()) #read the serial data and print it as line
    print incoming
    
    if 'Play/Pause' in incoming:
        pyautogui.typewrite(['space'], 0.2)
    if 'Rewind' in incoming:
        pyautogui.hotkey('ctrl', 'left')  
    if 'Forward' in incoming:
        pyautogui.hotkey('ctrl', 'right') 
    if 'Volume Incresaed' in incoming:
        pyautogui.hotkey('ctrl', 'down')
        
    if 'Volume Decreased' in incoming:
        pyautogui.hotkey('ctrl', 'up')
    incoming = "";