Gesture Controlled PPT Changer

What is the Project About

This is a device capable of recording and recognizing sophisticated 3d gestures and carrying out a preset action if the gesture matches, maybe changing the slide of a presentation, increasing volume or anything else that can be done using a keyboard and mouse.

For example: rotate the hand counter-clockwise to go to the previous slide & twist the hand to go to the next slide.

Some of its key features

• Can identify any sophisticated 3-Dimensional gestures.
• Recognise gestures using Dynamic Time Warping (DTW) algorithm.
• It communicates wirelessly to PC via Bluetooth, no need to connect any additional device to PC for communication.
• Has an OLED display and menu styled user interface for easy navigation.
• It is a standalone device, entire processing is done internally in Arduino Mega.
• It can remember gestures even after power is lost and retrieve after it is repowered again.

How it all started

It all started as fantasy if I could change the slides of the presentation with hand gestures instead of pressing those boring old buttons of a slide changer or a laptop.

Before actually building the device, I had some requirements for it

Requirements

• Should be as cheap as possible.
• Need to be small, portable and battery powered.
• All processing should be done internally, it should be a standalone device.
• Should communicate wirelessly, there should not be any additional receivers.
• Should be able to record custom gesture

Step 1: Parts Required

1. Arduino Mega - link
2. 0.96-inch I2C OLED display - link
3. MPU-6050 accelerometer gyroscope combo - link
4. HC-05 Bluetooth module - link
5. FTDI programmer - link
6. Perfboard - link
7. Push button * 2 - link
8. Battery - link

And some other common electronics components and tools such as soldering iron, USB cables etc...

Please use the above affiliate links to buy any of the above components, it will help in the future existence of the project.

Step 2: ​Configure the HC-05 Bluetooth Module As HID (Human Interface Device) Device

In this project, we need to send commands to the PC to change the slides, as we don't want any receivers to insert in the USB port of PC, we can proceed further with Bluetooth.

By default, the firmware which is present in the most common Bluetooth module, HC-05 can act as a slave only.

We need an HID (Human Interface Device) compatible Bluetooth module configured as a wireless Bluetooth keyboard to send commands to the PC to change slides.

We have two options for an HID device, one is to directly buy a HID compatible Bluetooth module RN-42 like this which costs 10x of ordinary Bluetooth module or to flash the firmware of RN-42 to HC-05, both of the modules are based on similar hardware.

I have learnt to change the firmware of HC-05 from Brian Lough and Evan Kale, they have done a wonderful job in explaining how to change the firmware, it makes no sense in reproducing the same here again, so I would like to leave links to their videos and give you the firmware files I have.

Find Brian Lough's video here

Find Evan Kale's video here

All the files required for firmware change can be downloaded from the google drive link here

Step 3: Building the Hardware

Now that you have an HID compatible Bluetooth module, you can start building the hardware.

Download the Fritzing circuit schematic file from here.

I recommend to build all the hardware on a piece of perf board and keep it as a shield for Arduino mega.

Step 4: Building the Software

This is the most important part of the project, the code.

Before actually opening my code, I would like to recommend you go to preferences and enable "Code Folding" if you are using Arduino IDE, now you can fold the parts of the code to get a clear idea of the code.

You can find the most updated code from my Github page.

There are many parts in the code, let me explain each part individually.

If you have any doubts, comment it down, I will be happy to help.

Step 5: Recording Gestures

The first step in the process of recognising gestures is to record them, and to record gestures, we are using accelerometer readings from MPU-6050.

The function that I have created for recording gestures is take_reading(), refer it in the code.

For gesture to be clear the sample size needs to be large and for faster processing, the gesture size needs to be small, I have found 50 to be a good fit for both the conditions. Now if we want to record longer gestures, we can average out every 2 or 3 elements to get gesture to 50 elements.

#define DOF 3	//3-degrees of freedom acc_x, acc-y, acc_z
#define avg_lenght 2 	//average out every 2 elements of reading
#define sample_size 50	

int reading[DOF][avg_lenght*sample_size];	//creating a 2-D array to store readings

//taking readings
for(i=0; i<avg-lenght*sample_size; i++)
{
  MPU6050.update();

  reading[0][i] = mpu6050.getAccX();	//taking readings of acceleration in g's, 1g, 1.2g
  reading[0][i] = reading[0][i]*50 + 50;   //ofsetting the value to 50
  if(reading[0][i]<0)	//limiting its value from 0 to 100 
    reading[0][i]=0;
  else if(reading[0][i]>100)
    reading[0][i]=100;

  //repeating the same for remaining 2 degrees of freedom, acc_y, acc_z
}

We have taken the readings, but these are not of sample_size elements, these are of sample_size*avg_lenght elements, we need sample_size elements, so we need to average it out.

if(avg_lenght>1)	//if we need to average
{
  for(i=0; i<DOF; i++)		//for each DOF
  {
    for(j=0; j<sample_size; j++)
    {
      for(k=0; k<avg_lenght; k++)
      {
        sum=sum+reading[i][avg_lenght*j+k];	//add every avg_lenght elemens
      }
      temp_values[i][j]=sum/avg_lenhgt;		//save avg of avg_lenght elements here
      sum=0;
    }
  }
}


else if(avg-lenght==1)	//no need to average
{
  for(i=0; i<DOF; i++)
  {
    for(j=0; j<sample_size; j++)
      temp_values[i][j]=reading[i][j];	//simply copying values to temp_values
  }
}<br>

Now readings for all DOF's are taken and saved to temp_values array.

Step 6: Saving the Recorded Gesture As Master Gesture

Now that we know how to record a gesture, we need a way to save that as a master gesture so that we can compare a gesture to it later.

The function that I have created for copying a gesture to master gesture array is copy_reading(from, to master, master _select), refer the code.

for(i=0; i<DOF; i++)
{
  for(j=0; j<sample_size; j++)
  {
    master[master_select][i][j] = temp_values[i][j];
  }
}

Now, we can record a gesture, save it as a master gesture.

Step 7: Saving Gestures to EEPROM​

We have saved the master gesture, but as soon as power is lost all the gestures saved till now will be lost. We need some way to save them so that we can retrieve them back again after the device is powered again.

We can use external I2C EEPROM's and attach these to Arduino, but this increases the cost, which we don't want.

We can indeed use Arduino's internal EEPROM to save gestures. For that, we need to include EEPROM.h library and we are ready to go.

The function to save master gestures to EEPROM is EEPROM_write().

int master_select;	//to select which master to select to EEPROM
//save 0-49 for master-0-x, 50-99 for master-0-y, 100-149 for master-0-z .....
for(i=0; i<DOF; i++)
{
  for(j=0; j<sample_size; j++)
  {
    addr=(master_select*sample_size)+(i*DOF)+j;
    EEPROM.write(addr, master[master_select][i][j]);  //write value of master to specific address<br>    delay(5);	//time to write to EEPROM
  }
}

The function to retrieve master gestures from EEPROM is EEPROM_read().

int master_select
for(i=0; i<DOF; i++)
{
  for(j=0; j<sample_size; j++)
  {
    addr=(master_select*sample_size)+(i*DOF)+j;
    master[master_select][i][j]=EEPROM.read(addr);   //save value from EEPROM to master gesture
    delay(5);
  }
<p>}</p>

Step 8: The Dynamic Time Warping Algorithm

Nov that we can record a gesture and master gesture, we need something to compare both, that's where DTW algorithm comes into play...

The function for DTW in the code is calc_DTW_score(), functions abs_sum() and Min() will be used in DTW calculations.

What is the Dynamic Time Warping (DTW) algorithm?

It is an algorithm to find how similar any two time-varying series are. It was initially developed for speech recognition purposes. For more details visit Wikipedia.

Higher the DTW_score, lesser the two time series are matching. For detection of matching the pair of a series, we can compare it to multiple series, with whichever series the DTW score is minimum, it is the matching series.

How to implement it?

I didn't find any simple library (that is suitable to run on 16 Mhz processors of Arduino) or code written for Arduino, so I need to build it. The whole algorithm that I have written is most simple I could get it to, and it is based on this video.

You can find only the DTW algorithm that I have implemented here.

The above picture is the matrix for DTW calculation, highlighted elements of the first row and first column are the elements we are comparing. the rest of the elements are required for calculation of DTW score.

cell  = difference of corresponding elements of arrays comparing + minimum of previously computed 3 values

As we are going to use difference and minimum of elements, I have created two separate functions "abs_sub" and "Min" for further use.

The elements (1, 1), (2, 1) & (1, 2) are just difference of corresponding elements of arrays, so the code for it is as below.

a[1][1]=abs_sub(a[1][0], a[0][1]);      //first element
a[2][1]=abs_sub(a[2][0], a[0][1]);
a[1][2]=abs_sub(a[1][0], a[0][2]);<br>

The remaining elements of the second row and second columns are difference + min of previously calculated 3 values, which is the previous element.

x=1		//first row remaining elements
for(y=2; y<matrix_size; y++)
  a[x][y] = abs_sum(a[x][0], a[0][y]) + a[x][y-1];

y=1		//first coulum remaining elements
for(x=2; x<matrix_size; x++)
  a[x][y] = abs_sub(a[x][0], a[0][y]) + a[x-1][y]

Now, the rest of the elements can be calculated

for(x=2; x<matrix_size; x++)
{
  for(y=2; y<matrix_size; y++)
  {
    a[x][y] = abs_sub(a[x][0], a[0][y]) + Min(a[x][y-1], a[x-1][y], a[x-1][y-1])
  }
}

Now that we have calculated the entire DTW matrix, we can now calculate the DTW score, by adding up the minimum of the next 3 elements from the bottom right to top left, see the image above for more clarity.

x = sample_size;	//moving to bottom right
y = sample_size;
DTW_score = a[x][y];	//sarting adding from there


while(x!=0 && y!=0)	//till top left is reached
{
  if(a[x-1][y-1]<=a[x][y-1] && a[x-1][y-1]<=a[x-1][y])		//if diagonal is minimum
  {
    DTW_score = DTW_score + a[x-1][y-1];	//add it
    x--;	//go to its position
    y--;
  }
  else if()    //if top is minimum
  {
    DTW_score = DTW_score + a[x][y-1];
    y--;
  }
  else 		//add its side
  {
    DTW_score = DTW_score + a[x-1][y];
    x--;
  }
  //repeat until top left is reached
}


return DTW_score;	//done calculating DTW score

The calculation of DTW_score is completed when the top left is reached. Now this score can be used to compare how similar any two series are.

Step 9: Adding an Action to Do If Gesture Match

We now have DTW algorithm to compare a gesture to several master gestures, we need to set actions to do if the gesture matches with a master gesture.

The function for this in the code is do_action(int a).

You can do all the things that you can do with a keyboard, maybe increasing the volume of video playing or do an action with multiple key presses (ctrl+shift+esc) to turn on task manager, maybe launching an application by using Autohotkey and many more.

You can enter all commands by using this USB HID table.

For entering a number, character or a phrase, you can use Bluetooth_HID.write / print("some character or phrase"); and for entering a key from USB HID table, you need to use Bluetooth_HID.print("something", HEX);

Some commands in PowerPoint Presentation

n - go to next slide

p - go to the previous slide

w - display white screen

b - display black screen

if(a==0)	//do action corresponding to master 0 gesture
{
  bluetooth_HID.write("n"); 	//send keystroke n from HID Bluetooth module as sent from wireless keyboard
}
//actions for remainging masters

Step 10: Creating a User Interface

We have all the necessary functions for recognising a gesture and additional, now we need a User Interface to access all of the functions.

If you don't want to create a User Interface, you could go with the most simple code I have created and modify it according to your need.

Here I have created User Interface in the void loop...

The first step is to initialize the display by cleaning it and setting the text size and text position

#define display_init display.clearDisplay();display.setTextSize(1);display.setTextColor(WHITE);display.setCursor(0,0)

after this, we can display the desired text on the display

while(1)
{
  display_init;	//initialize display
  
  display.println("What?");
  display.println("  a. test gesture");
  display.println("  b. record master");
  display.println("  c. update EEPROM");
  display_set_cursor(line);	//function we have created to display a cursor on selected line
  display.display();	//update the screen

  //press down_button to move cursor down
  if(down_buton_pressed)
  {
    delay(10);	//to remove button bouncing
    line++	//go to next line
    if(line>2)	//if reached to last line
      line=0;	//go to first line
  }

  //press select_button to select option where cursor is present
  if(select_button_pressed)
  {
    delay(10); 		//to remove button bouncing
    break;	//go to next lines by crossing while(1)
  }
}

if(line==0)	//if option on line 0 is selected, to test gesture
{
  display_init;	//initialize display
  display.println("recording gesture");
  display.display();<br>  digitalWrite(13, HIGH);

  take_reading();

  digitalWrite(13, LOW);
  display_init;
  display.println("done recording gesture");
  display.display();

  for(i=0; i<num_masters; i++)		//calculate DTW_score for each master
    DTW_score[i]=calc_DTW_score(temp_values, master, i);
  
  min_score=DTW_score[0];	//finding minimum of all DTW_scores
  for(i=1; i<num_masters; i++)
  {
    if(DTW_score[i]<min_score)
      min_score=DTW_score[i];
  }

  for(i=0; i<num_masters; i++)	//finding minimum is which master
  {
    if(min_score==DTW_score[i])
      break;	//exit from for loop
  }

  display_init;
  display.println("master is: ");
  display.println(i);		//display master on screen
  display.display();
  do_action(i);		//do corresponding action to master gesture
  delay(2000); 	//display on screen for 2 seconds
}

else if(line==1)	//similarly for line 1 and line 2
{
}

else if(line==2)
{
}

Done!