Introduction: Gesture/Pattern Recognition Without Camera : TOF Sensor & SPAD Array

By Lab_1314 Channel
About: Born in the Alps, I love travelling, paragliding, trekking and, of course, designing projects. :) I learned a lot from open source project on the last few years, i am very happy to inspire new people with my w…

UPDATE : 12/11/2021 : New video Online :

Grumpy Hedgehog is a connected device that allows you to read and interact with movements. Its LCD screen displays the Hedgehog's expressions according to his understanding.

We can easily create our own movements and patterns to transmit actions through several communication channels (USB HID, Wifi, IR...) or simply use its relay and servomotor.

Supplies

  • Arduino MKR 1010 (any Arduino work for most parts)
  • VL53L1X TOF Board
  • TFT LCD 1.44
  • SG90 Servo Motor
  • TSOP2138 Receiver
  • DS18B20 Temperature
  • Single Relay Board

Github REPO

Step 1: What Problem Are We Going to Solve?

Today we are used to have physical contact with the objects that surround us. This applies to switches, touchscreens, remote controls... The current crisis forces us to question these habits and to enter a new digital era where physical contact with certain objects must be reduced to a minimum.

While speech recognition allows a complete approach for transmitting information, it is still quite complex for machines and humans to grasp. A child is usually able to understand and imitate many signs long before his or her first words.

Why not use signs to create simple interactions that can convey more information than ON/OFF?

Once you send a movement or a pattern, GrumpyHedgehog can :

  • Control a computer by sending keyboard command (movie pause, volume...)
  • Track the number of person in a shop on your smartphone
  • Safely unlock a computer
  • Send encrypted informations to a server
  • Control light or any power components
  • Send IR command to your TV
  • Move a servo motor

Step 2: Key Elements

In addition to offering a modular solution with its different interfaces Grumpy Hedgehog is particularly distinguished in 4 aspects:

  • It uses a VL53L1X Time Of Flight Sensor allowing complex motion recognition with 16x16 SPAD sensing array, it can run on most micro-controllers such as SAMD21 (MKR Wifi 1010) and Atmega328p (NANO/UNO board).
  • The second aspect is the management of the LCD screen which is optimised to store in FLASH and display a moving image. The first tests on my Arduino UNO are promising!
  • The third differentiating aspect is its design. The challenge here is to create a decorative object that you will gladly leave on display in your living room or office.
  • It uses the ATECC508 secure element to create a strong security layer based on Public/Private Key to authenticates node IDs, ensures the integrity of messages, and supports key agreement to create session keys for message encryption. (This part isn't fully finished)

So you will understand that the GrumpyHedgehog with his thirst for learning, his gestural communication, his cute look and his grumpy character will quickly become your new companion.

Step 3: How It Work

By default GH detects three movements: Left, Right and Up. The left and right movements allow you to send commands depending on the actual mode:

To switch from one mode to another, simply leave your hand within 15cm of the sensors for 3 seconds. You don't have to use all the mode, just comment the modes you are not using on the top of Arduino Program.

In addition to the different modes, you can always use Pattern Recognition with the upward movement the upward. GH will detect on a 4x4 pixel grid the shape of the object in front of it. The Arduino program allows to easily add Pattern and match it to an action.

To add security, a Pattern and a Keyboard Password can be saved in the secure element ATECC508. With the right configuration, it will be impossible to recover the Password unless the Pattern match. See the ATECC508 section for more informations. (This part isn't finished yet)

Step 4: Time of Flight Sensor and Algorithm

The use of VL53L1X TOF sensor is really interesting. But what makes it different from many competing proximity sensing technologies? (Ultrasonic, Sharp IR angle, LIDAR...)

Beyond the fact that it allows a detection up to 4 meters, a 50Hz refresh rate and a good immunity to light and surfaces reflexion. What I was particularly interested in is called SPAD Array (Single Photon Avalanche Diode) and ROI (Region Of Interrest).

This is because VL53L1X not only detects proximity, it can also target this information to a detection field. The SPADs thus form a 16x16 pixel grid like this :

There is however some restrictions, the detection area cannot be less than 4x4 pixel. By using a reading of each area without overlapping, one can therefore have a reading of 4 pixels by 4 pixels, i.e. 16 detection zones. The other restriction is speed. Using 16 pixels at 50Hz per reading is barely 3Hz for a full reading. You can find an Arduino/Processing Example on Git repo.
The mode reading Pattern read the 16 ROI and print everything on the LCD.

After several tests, the configuration that seems optimal to detect 3 movements (left, right, up) is with 3 ROI. As soon as a SPAD detects an activity, it trigger a detection_flag to 1. The last SPAD to detect something while this detection_flag is active considers it as a movement in this direction.

In theory we could also do a DOWN motion detection with this pattern. But my algorithms are not yet robust enough and the refresh rate in relation to the motion doesn't make it easier.

To further optimise, I improved the detection by reading all the SPAD at 50Hz. When it trigger something, then I set a detection_flag to 1 and then I start reading my 3 ROI at 17Hz.

There is a lot of testing to be done to optimise ROI for movements detection. If you have any advice, please comment or send me a message.

Step 5: LCD, Emotions and Optimisation

One of the challenges on this project is to run on both SAMD21 and Atmega328p. With some modules like TOF or relay, this won't be too much of a problem, the LCD is here a real problem for 2 reasons:

Problem 1 : Memory

With its 32kb of Flash, the Atmega328p can't store a lot of information onboard. Let's imagine that we want to store an image encoded on a RGB565 (2 bytes) of 128x128 pixels. We have : 128x128x2 = 32768 bytes. A single image on a small screen would already completely overload the memory!

Problem 2 : Refresh Rate

To refresh this screen of 16384 pixels it will be necessary to pay also attention to speed. The Atmega328p has only one SPI. If you want to use, for example, an SD card to overcome the memory problem. SD and LCD will be on the same bus. So you will have to do the same on the SPI: get the image from the SD and send it to the LCD...
I found someone who have a solution to this problem HERE, but it implies a RAW storage of the images, which can be quite difficult to stay flexible.

Implementation

To start with, I imagined how to represent my hedgehog with simple shapes that I could generate with the Arduino GFX library. Here, my screen displays the eyes (2 circles) and the nose (1 circle + 3 mini circles) of my hedgehog. I can thus simply use GFX without having to load an image in memory. In addition, I can draw the circles without reloading the whole screen, which allows me to gain a lot of speed.
In order to validate the shapes and movements, I started by a simulation on Keynote

****GIF Missing keynote expression

On the LCD, to implement a movement simply, I redraw the previous shape in black and then I redraw it with color to its new position.
On the Arduino MKR 1010, I can take advantage of the speed and the extra memory to make my animations more fluid and complex.

UNO = 16MHz ---------- MRK 1010 = 48MHz

UNO = 8 bit instruction ---------- MKR 1010 = 32 bit instruction

UNO = 32Kbit Flash ---------- MKR 1010 = 256Kbit Flash

Step 6: HID Keyboard

Unlike an Arduino Uno, USB port on MKR 1010 is connected directly to the main micro controller. This feature, also present on the Arduino Leonardo, offers the possibility to use USB-HID implementation. This allows your project to behave like a keyboard or a mouse. UNO Board use a second micro controller (Atmega32u2) to transmit data. We can't natively use USB HID when we program the Atmega328p.

From Arduino MKR 1010 Schematic

In our case, we will be able to use this specification to send keyboard commands directly to a computer to manage different functions (volumes, movie pause, next episode shortcut...).
Another interesting feature is the presence of PA18 on the 4th PIN of the USB OTG. The activation of this PIN should allow us to reverse the USB master/slave behavior.

After a first test on last version of SAMD library (1.8.6). The keyboard wasn't working.It was recognised as a COMport... I found this tutorial explaining that if you downgrade to version 1.6.21 it will work. And it's true :-)

Step 7: Wifi Synchronisation With Blynk for Person Counter

To facilitate this part, I used the BLYNK environment to manage the display of data on smartphone. Blynk allows you to connect a module to a web interface or an application with great ease.

With a limited number of lines of code, one or more objects can be connected to a graphical user interface. It can be used on IOS or Android via a simple TOKEN exchange that will allow to make the link between application and objects.

In WIFI mode, we configure the VL53L1X to extend its range to 4 meters and count the number of people passing in front of it. If it detects movement to the left, it will send a notification to decrement the number of people and the opposite to the right. Thus, with a flow respecting 1 meter of regulatory distance, the BLYNK interface will indicate in real time the number of people inside a room.

This application can also be used with several GH to monitor several inputs/outputs. Here is an example :

Step 8: Safe Element Storage I2C ATECC508

In order to protect its data, the Arduino MKR 1010 implements an ATECC508. With a good implementation, this small device allows the integration of almost infallible security in the communication. Check this very helpful Intructable on the subject.

After some research, I found 2 Arduino libraries (ArduinoECCX08 and Sparkfun) allowing to integrate some functions of the ATECC508. WARNING: the configuration of the device is permanent. If you accept the default configuration of these libraries, you will never be able to change it again.

There are many ways to use this module, but these libraries focus on generating Public/Private key pairs and certificates to ensure the integrity of messages. There are already several tutorials to use these functions, I tested this one which worked well to connect to Google Cloud IOT with my Arduino MRK 1010. There is also this example to connect to Azure IOT Hub (but I didn't test it on MKR).

In our case, we would like to use ATECC508 to store private information (a pattern and a password). The SAMD21 would have to send Patterns to it and the ATECC508 would have to compare them with the one in memory. If this one matches, ATECC508 will send the password. (This part isn't finished yet !)

To integrate this, you must already understand the notion of configuration. Fortunately, the ATECC508 Datasheet is available on the internet (which is not the case for all secure elements, check for the complete datasheet with 109 pages). What we need is to set the correct configuration in "SlotConfig" (Section 2.2.1 SlotConfig (Bytes 20 to 51)) to enable "CheckMac" command (See section 3.2.7 Password Checking).

To use Password checking we need to configure ReadKey in the target SLOT to 0 (bit 3-0 of SlotConfig). This enable "CheckMac/Copy". If we check the default configuration of ArduinoECCX08 we have :

SLOT<0> : 0x83, 0x20, // External Signatures | Internal Signatures | IsSecret | Write Configure Never. This mean we have 0011 (0x3) as ReadKey instead of 0. So we need to configure a Slot with 0xX0, 0xXX

On datasheet we have "If the password is to be mapped to a secondary value (using the third option above), then the target slot containing this value is located in the next higher slot number (i.e. the password’s slot number plus one);"

So we can use

  • SLOT<8> with 0x80, 0x00 to store the patern
  • SLOT<9> with 0x80, 0x00 to store the secret password

The SAMD21 and the ATECC508 communicate with SPI on PA8 (SDA) and PA9 (SCL) or with One Wire with PA8 (SDA). We will use the SPI as it's the default configuration.

From Arduino MKR 1010 Schematic

Step 9: Relay 220V / Servo Motor and Temperature Sensor

For this part, I have created a small outer box allowing me to isolate these components from the main system. Remember that the Servo and Relay work better at 5V, that's the reason I used a Step-Up board to convert my signals voltage. The 3D is on the GIT repository.

Step 10: IR

GH can also learn signals from your IR remote control and send them to your TV. At the moment, I haven't had time to link this learning function to my system. So it is necessary to use a third party program to do the learning and then copy the frames to be sent to the main program. This part isn't finished yet. Few more days ! The learning program is also on the GIT Repository.

Step 11: Energie Optimisation

The overall design of GH is currently not optimised to fit on a battery. It uses different components that consume a lot of energy, such as a 220V relay, a servomotor, an LCD and also WIFI. Its global consumption will probably be around 150mA ! So I do not find this project relevant in its entirety to be used on battery.

However, GH has been built in a modular way, allowing it to easily remove/modify certain modes. Here is a more minimalist configuration that would allow me to respond to the "Counting Person" usage that would allow me the most adequate :

  • Arduino MKR 1010 with sleep mode and BLE configuration
  • VL53L1X using optimised sleep mode and interruption
  • LCD screen turned OFF when nothing is detected
  • Lipo Battery 2000mA

With this configuration, consumption could drop to a few mA and allow several months without recharging.

Step 12: Build the Device

Block diagram

Hardware

Software

You can find all Software on the Github repository of the project. Here are the librairies I used from the Arduino libraries manager :

  • Adafruit_GFX.h & Adafruit_ST7789.h
  • SPI.h
  • Keyboard.h
  • SparkFun_VL53L1X.h
  • Wire.h
  • WiFiNINA.h
  • BlynkSimpleWiFiNINA.h
  • ArduinoECCX08.h

3D Design

I admit I've already drawn many 3D on Sketchup, but I really didn't know how to do this one... After some research, I went on a tube shape in which I drew circles. By deforming these circles I was able to arrive on the shape I was looking for. To adjust a dimension based on the center of the shape, simply hold down the ctrl key.

For the front, I just placed a flat face using the "Intersect Faces" option. Here is the result :

0

And here is the rendering on Sketchup with a little color and a lot more spines

And ready to print with CURA (some spines are missing...)

Step 13: Conclusion and Improvement

I really enjoyed working on this project. The SPAD technology used by the VL53L1X is interesting and the integration of the LCD screen in the 3D printing quickly gives nice results.

Nevertheless, I still have a lot of things to improve:

  • Changing the LCD screen: with more pixels for a nicer rendering
  • Redesigning a 3D to facilitate the positioning of the MKR 1010 card and better integrate the screen
  • Finish the program for IR emission and reception
  • Finish the Password ATECC508 program
  • Build a low power version on battery (remove Relay, Servo and Wifi. Add Bluetooth).
Automation Contest

Second Prize in the
Automation Contest