HackerBox 0076: Biometrics

Introduction: HackerBox 0076: Biometrics

Welcome to HackerBox 0076. The adventure into biometric identification leverages face recognition, fingerprint scanning, and voiceprinting. The hardware used includes the ESP32 dual-core microcontroller with Wi-Fi, an OV2640 camera, an ultra-slim fingerprint scanner, and a MEMS microphone. TensorFlow Lite for Microcontrollers and similar software tools implement artificial intelligence and machine learning algorithms using neural network techniques for natural data modeling.

HackerBoxes is the monthly subscription box of electronics, computer technology, and hacker culture. Each HackerBox is a discovery box, which means all members await and enjoy a new surprise each month. Tech, toys, knowledge, and fun... It's like having a hacker convention, your birthday, and the first day of school - every month - right in your mailbox.

There is a wealth of information for current and prospective members in the HackerBoxes FAQ. Almost all of the non-technical support emails that we receive are already answered there, so we'd really appreciate it if you can take a few minutes to read the FAQ.

Supplies

This Instructable contains information for getting started with HackerBox 0076. The full box contents are listed on the product page for HackerBox 0076 where the box is also available for purchase while supplies last. If you would like to automatically receive a HackerBox like this right in your mailbox each month with a $15 discount, you can subscribe at HackerBoxes.com and join the party!

A soldering iron, solder, and basic soldering tools are generally needed to work on the monthly HackerBox. A computer for running software tools is also required. Have a look at the HackerBox Workshops for basic tools and a wide array of introductory activities and experiments.

The most import thing you will need is a sense of adventure, hacker spirit, patience, and curiosity. Building and experimenting with electronics, while very rewarding, can be tricky, challenging, and even frustrating at times. The goal is progress, not perfection. When you persist and enjoy the adventure, a great deal of satisfaction can be derived from this hobby. Take each step slowly, mind the details, and don't be afraid to ask for help.

Step 1: Biometric Identification

Biometric identifiers are the distinctive, measurable characteristics used to label and describe individuals. Biometric identifiers are often categorized as physiological characteristics, which are related to the shape of the body. Examples include, but are not limited to fingerprints, vein structures, facial recognition, DNA, hand geometry, iris recognition, retina, and scent. Similar, behavioral identifiers are related to the pattern of behavior of a person, including but not limited to typing rhythm, gait, signature, behavioral profiling, and voice.

Traditional means of access control general include token-based identification systems, such as a driver's license or passport, and knowledge-based identification systems, such as a password or personal identification number. Since biometric identifiers are unique to individuals, they are more reliable in verifying identity than token and knowledge-based methods; however, the collection of biometric identifiers raises privacy concerns about the ultimate use of this information.

(Wikipedia)

Step 2: Programming the ESP32 Microcontroller

The ESP32-CAM Development Module does not include a USB bridge or USB connector, so connecting external USB support is required. In this case, we'll use the Micro CP2102 Serial USB Module (datasheet).

Solder the six pin header onto the Micro CP2102 Serial USB Module.

The OV2640 camera can be left off of the ESP32-CAM module for now.

Use five female-female jumper wires to connect the ESP32-CAM to the CP2102 USB Module as shown here.

Install the Arduino IDE software and set up the ESP32 board support package. This tutorial provides detailed instructions for that process. Currently, version 1.0.6 of the ESP32 package is preferred for the projects and libraries referenced in this Instructable. If a 2.x version is installed, it may be helpful to roll back.

In the IDE, select:

Tools > Board > ESP32 Arduino > AI Thinker ESP32-CAM

Also select the Tools > Port entry that appears when the CP2102 is connected to the PC

Be sure to have the "Programming Mode" jumper wire between I/O 0 and GND

Load the blink sketch: File > Examples > 01.Basics > Blink

Replace LED_BUILTIN with the value 4 in three different locations within the sketch.

Hit the arrow icon in the IDE to initiate compile and upload to the ESP32-CAM target.

If compiling completed but they upload is waiting for the bootloader with "....._____....._____" in the status window, hit the RST button on the ESP32-CAM. That should cause the upload to start.

Once the upload completes with "hard reset":

Remove one end of the "Programming Mode" jumper wire and hit the RST button on the ESP32-CAM.

This sketch will run, blinking the Camera's FLASH LED (connected to I/O pin 4).

Don't look directly at the FLASH... It's extremely bright.

This sequence needs to be used every time the ESP32-CAM is programmed:

  1. connect "Programming Mode" jumper wire
  2. press the IDE upload arrow icon
  3. hit the RST button if upload is waiting "....._____....._____"
  4. wait for upload to complete with "hard reset"
  5. disconnect the "Programming Mode" jumper wire
  6. open the serial monitor and set baud rate (when used)
  7. hit the RST button


Step 3: Ultra-Slim Fingerprint Scanner

The ultra-slim fingerprint scanner features a tiny package usually found embedded within mobile devices such as smartphones or laptops. The scanner is based on an AS608 SOC from Synochip. The AS608 series chips feature high-performance 32-bit micro-processors based on the ARM Cortex-M. The chips feature a working frequency of 144MHz, 4KB of cache memory, 512KB of embedded Flash, 128KB SRAM, and extendible external SQI Flash up to 16MB.

Wire up the ultra-slim fingerprint scanner as shown using the included breakout cable. As usual, the colors of the wires are not necessarily meaningful. Instead, mind the relative pin ordering of the connections. Note that five of the breakout wires plug directly to the ESP32-CAM while one plugs onto the 3V3 pin of the Micro CP2102 Serial USB Module.

In the Arduino IDE, use Tools > Manage Libraries to search for and install two separate libraries:

  1. EspSoftwareSerial (by Dirk Kaar)
  2. Adafruit Fingerprint Sensor Library

Enroll (Learn) Fingerprints

Open: File > Examples > Adafruit Fingerprint Sensor Library > enroll

Make the following three modifications to the sketch...

Under the line to include <Adafruit_Fingerprint.h>, paste this line in :

#include <SoftwareSerial.h>

replace the line #define mySerial Serial1 with:

SoftwareSerial mySerial;

replace the line finger.begin(57600) with:

mySerial.begin(57600, SWSERIAL_8N1, 14, 15, false);

Compile and upload the sketch.

Before disconnecting the PROGRAM jumper wire and hitting rest,

Open the Serial Monitor and set it to 9600 baud

The interface printed to the Serial Monitor allows selection of ID # 1-127 to store separate fingerprint models.

Evaluate Fingerprints

Open: File > Examples > Adafruit Fingerprint Sensor Library > fingerprint

Make the same three modifications used for the enroll sketch.

Compile, upload, and reset.

Test the enrolled fingerprints and observe the output on the Serial Monitor.

Also try other fingerprints that were not enrolled.

How effective are the models created by the enrollment process to match the same fingerprints later?

How effective are the models to exclude false matches to other fingerprints?

Step 4: Machine Learning

Machine Learning is the process of taking in data (fingerprint sensor image) and associated answers (that image is my index finger) to create a representative model (learning) that can be used later (evaluated) to determine if new data (a new fingerprint sensor reading) is likely to be associated with the same answer (my index finger). The model is the key because it represents the learning obtained from the training process. In the fingerprint example, "enrollment" is how we train the models.

Step 5: Omnidirectional MEMS Microphone Module

The INMP441 (datasheet) is a omnidirectional MEMS (micro electro mechanical system) microphone. The INMP441 provides high-performance, low power, digital-output, and a bottom-side sound port. The complete INMP441 solution consists of a MEMS sensor, signal conditioning, an analog-to-digital converter, anti-aliasing filters, power management, and an industry-standard 24-bit I²S interface. The INMP441 has a high SNR (signal to noise ratio) and a flat, wideband frequency response.

Solder and Connect Header Pins

Solder the six supplied header pins onto the microphone module. The header pins should extend from the same side of the PCB as the metal microphone housing. Sound enters the microphone from the other side of the PCB through the small gold circular opening.

Wire up the INP441 module as shown using five female-to-female jumper wires.

Record Audio via Microphone Module

Download ESP32_INMP441_RECORDING.ino from this github repo.

This sketch will record sound from the INMP441 microphone into a wave file. The wave file is created in the SPIFFS (Serial Peripheral Interface Flash File System) of the ESP32-CAM module. SPIFFS is a lightweight filesystem created for microcontrollers, such as the ESP32, that have a flash chip connected via the microcontroller's SPI bus.

Update the three lines of the sketch that define the I/O pin connections, like so:

#define I2S_WS 13
#define I2S_SD 14
#define I2S_SCK 16

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor to observe the progress of recording the wave file to the SPIFFS. This is where you provide your spoken words (or other sounds) to the microphone.

Retrieve Recoded Wave File from ESP32

Open the File System Browser sketch:

File > Examples > WebServer > FSBrowser 

Update the SSID and Password to a 2.4GHz Wi-Fi network that your PC also has access to:

const char* ssid = "wifi-ssid";
const char* password = "wifi-password";

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor to observe the IP address obtained by the ESP32.

Enter "192.168.168.168/recording.wav" in the URL field of your browser but replace the actual IP address from the Serial Monitor. This should either play the stored wave file from the ESP32 or cause the wave file to be downloaded to the PC where it can be played from download folder.

Obviously, the wave file should playback whatever speech (or other sounds) that you recorded from the microphone into the SPIFFS.

Recording, storing, and playing back sound may seem like a simple series of tasks. However, we can see from this exercise that there is quite a bit to it - especially with a small microcontroller. It can be done though, so let's keep in mind that the devices may be listening.

Step 6: TensorFlow Lite Voice Models

TensorFlow Lite for Microcontrollers is designed to run machine learning models on microcontrollers and other devices with only few kilobytes of memory. The core runtime just fits in 16 KB on an Arm Cortex M3 and can run many basic models. It doesn't require operating system support, any standard C or C++ libraries, or dynamic memory allocation.

In the Arduino IDE, use Tools > Manage Libraries to search for and install the library:

TensorFlowLite_ESP32

Open the example sketch...

File > Examples > TensorFlowLite_ESP32  > micro_speech_ESP-EYE

Switch to the tab for file "audio_provier.cpp" and adjust the I/O pin definitions, like so:

#define I2S_PIN_CLK    16
#define I2S_PIN_WS     13
#define I2S_PIN_DOUT   I2S_PIN_NO_CHANGE
#define I2S_PIN_DIN    14

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor and set it to 115200 baud to observe the sketch identifying utterances of "YES" and "NO".

The TensorFlow models used in the sketch were trained to apply to generic voice patterns using specific microphones and capture equipment. How accurate are they?

If you were to train your own TensorFlow models with your actual voice and audio hardware, using those models should be more accurate and possibly more sensitive to your specific voiceprint. Like a fingerprints, voiceprints can attempt to identify one particular person and exclude most, or all, others.

Here is more information on training your own models if you'd like to give that a try.

Step 7: Face Detection and Recognition

Install the OV2640 camera into the ESP32-CAM module. The camera connector on the ESP32-CAM is a white slot with a black tab. The black tab hinges forward and down away from the PCB and towards the white portion of the connector. This opens the camera connector. Once that is opened, the camera's flex connector can be inserted into the white slot with the lens facing out. Finally, the black tab is pressed back down up into the slot connector. Note that the lens has a protective cover sheet than can be peeled off prior to use.

In the Arduino IDE, open the sketch: 

File > Examples > ESP32 > Camera > CameraWebServer

Update the sketch with the SSID and Password for a 2.4GHz Wi-Fi network that your PC also has access to:

const char* ssid = "*********";
const char* password = "*********";

In the section of the sketch titled, "Select camera model", comment out the default line and uncomment the line for CAMERA_MODEL_AI_THINKER.

Compile and upload the sketch.

Reset the ESP32.

Open the serial monitor and set it to 115200 baud to observe the IP address.

Open that IP address in a browser and begin streaming from the webcam.

Experiment with the Face Detection and Face Recognition functionality.

How effective are they?

Theory of Operation

Here is a very nice tutorial on Additive Angular Margin Loss (aka ArcFace) which is the algorithm employed by the ESP Human Face Recognition Model.

Step 8: Analog Computers

Will A.I. usher in a new era of analog computing?

Step 9: Expect Us

We hope you are enjoying this month's HackerBox adventure into electronics, computer technology, and hacker culture. Reach out and share your success in the comments below or other social media. Also, remember that you can email support@hackerboxes.com anytime if you have a question or need some help.

What's Next? Join the party! Live the HackLife! Get a cool box of hackable gear delivered right to your mailbox each month and enjoy a generous member discount. Surf over to HackerBoxes.com right now and sign up for your monthly HackerBox subscription.

2 People Made This Project!

Recommendations

  • Origami Speed Challenge

    Origami Speed Challenge
  • Trash to Treasure Contest

    Trash to Treasure Contest
  • Science Fair Challenge

    Science Fair Challenge

16 Comments

0
eburman
eburman

Tip 2 months ago

Remember to disconnect the program mode jumper!!!
I seem to habitually forget. Then I scratch my head wondering why the damn thing won't work. I end up double, triple and quadruple checking my wiring. I waste more time trying a number of other trouble shooting tasks and then about 15-20 minutes later I finally realize that I forgot to pull the jumper. Doh! So don't be like me.

0
ftlfrm
ftlfrm

Reply 18 days ago

Haha I had the exact opposite issue. I kept forgetting to re-plug what I pulled when I went to compile again.

0
grow23
grow23

2 months ago

So steps 8 and 9... am i missing something?

0
vleon42
vleon42

Reply 5 weeks ago

Step 9 is a reference to HBO's TV series Westworld, it does feels like this one is a bit unfinished yes.

0
eburman
eburman

Tip 2 months ago

I was having difficulty getting the "micro_speech_ESP-EYE" to function. The serial monitor was just cycling through a continuous loop of error messages. I mostly had no idea what they meant but I noticed something about "DIO." The default Flash Mode set in Board Manager is "QIO" so I changed the Flash Mode to "DIO". That worked. It all seems to work as it should now. It recognizes verbal "yes" and "no" inputs through the mic.

0
vleon42
vleon42

Reply 5 weeks ago

Thanks for this. Found more explanation here: https://www.esp32.com/viewtopic.php?p=5523&sid=3f6...

tl;dr: QIO is using 4 GPIOs to transfer data (faster) but for some reason it seems to leave them in a bad state and when running a program that uses GPIO16 it then crashes.

0
vleon42
vleon42

5 weeks ago

I had some issues where the ESP32 would crash whenever GPIO16 was used (eg: recording from mic), but was able to get around it by either swapping it with GPIO15 or change Flash Mode to "DIO" as @eburman pointed out.

Botched the soldering on the mic, and some solder fell into the mic hole, so my recording quality is pretty terrible -_-

I was able to get the camera to work, but whenever I try the face detection function, I get a corrupt heap error.

heap.PNG
0
dstockto
dstockto

2 months ago

I am having a good deal of trouble with some parts of this one. My first issue is with recording from the microphone. I am getting a "Guru Meditation error" which appears to be something related to not being able to read or write from the SD card. So that sort of put a damper on being able to complete recording or playback/download.

The next issue I've got is that I am unable to get the camera to be recognized. "Camera probe failed with error 0x20004". Looking through google searches there was some talk about bringing the PWDN_GPIO_NUM high "for awhile" and then down "for awhile". I tried adding to to do this but I'm still not able to get the camera to work. I've even tried powering from a power bank and from bench power in case I wasn't getting enough juice, but that made the serial response be junk and didn't appear to make anything work anyway. Any help on this would be much appreciated. The fingerprint scanner worked great and was pretty cool. I'd really like to be able to use the mic and the camera too.

0
ebchopra
ebchopra

Reply 2 months ago

I had the same error with the camera. After some troubleshooting I realized that I had the wrong camera model selected:

// Select camera model
//#define CAMERA_MODEL_WROVER_KIT // Has PSRAM
//#define CAMERA_MODEL_ESP_EYE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_PSRAM // Has PSRAM
//#define CAMERA_MODEL_M5STACK_V2_PSRAM // M5Camera version B Has PSRAM
//#define CAMERA_MODEL_M5STACK_WIDE // Has PSRAM
//#define CAMERA_MODEL_M5STACK_ESP32CAM // No PSRAM
//#define CAMERA_MODEL_M5STACK_UNITCAM // No PSRAM
#define CAMERA_MODEL_AI_THINKER // Has PSRAM
//#define CAMERA_MODEL_TTGO_T_JOURNAL // No PSRAM

Don't know if this helps...

0
dstockto
dstockto

Reply 2 months ago

I did have that part done, but it still does not detect the camera.

0
eburman
eburman

Reply 2 months ago

Did you remember to also comment out the other camera? I selected the AI_Thinker model but forgot to comment out the WROVER_KIT, so both camera's were selected and of course nothing worked. Once I commented out the WROVER everything worked fine for me.

0
LloydT14
LloydT14

2 months ago on Step 3

The wire colors on the AS608 connector that I received are DIFFERENT than the ones shown in the illustration. If you get a "Did not find fingerprint sensor :(" error, please follow the positional wiring diagram rather than the color coded one. The sensor should light up blue if the wiring is correct.

0
eburman
eburman

Reply 2 months ago

Yup! Same with mine. The White and Black wires were swapped. For my AS608 - white goes to GND and black goes to IO15. Thank you for mentioning it! Spared me some headaches.

0
GPop71
GPop71

Reply 2 months ago

Good call on the lighting up blue (pulsing blue actually). The wire colors didn't trip me up after seeing your comment, but I was off by one pin at first!

0
ebchopra
ebchopra

Question 2 months ago on Step 7

Everything seems to work except the facial detection/recognition. The streaming works fine, but nothing happens when I turn on the facial detection sliders or click the "enroll face" button. Is there something I should be looking for?

0
Tsegner
Tsegner

Answer 2 months ago

Same here