Introduction: Candy Dispenser With Face Recognition

This candy dispenser is using neural networks to recognise people's faces and give candy to familiar people.

Its "brain" is ESP32 microcontroller, so it can be easily programmed to follow any logic we want it to, e.g. it can give unlimited candy to person A, only 5 a day to person B and only 1 or none to any guests.

Of course, this isn't full-proof security system and one can easily untwist the top lid and grab as many candy as he likes. Instead, intention here was to make an eye-catching and original "smart" device and explore facial recognition technology in the process - and to have fun of course!

Basically, it's a way to make indulgence in sweets even more fun and addictive ;)

Enjoy!

Step 1: Supplies

  1. Battery Powered Candy Dispenser
  2. ESP32-CAM
  3. OV2640 75mm Camera
  4. OLED Display
  5. Relay
  6. LDR Photoresistor
  7. Resistor 10K
  8. Wires
  9. FTDI Serial Adapter (for programming)
  10. Polarized Capacitor 16V 330uF (optional)

Step 2: What It Does

This project significantly grew beyond simple "Hey I know you, take a candy" case, so it might take you some time to digest everything that is going on in here.

On the other hand expanding beyond the very simplest case, presented a number of challenges I had to work on and I've learned a lot in the process. All that might be useful to someone else or at least spark an idea for some other project. At least that was the case when I looked at other people's projects. Hopefully this will help someone too...

Anyway, here's a list of things this candy dispenser does:

  • Constantly monitors and looks for faces; tries to match detected face with a list of familiar/stored faces.
  • Allows faces to be stored (or deleted) in the permanent flash memory - currently two (primary and secondary)
  • It detects "hand proximity" event and it dispenses candy if all other criteria is met (face is recognised, it hasn't expired, it is a familiar face etc.)
  • It interacts with a user via OLED display
  • It has several logging modes and can write logs to serial monitor, display, MQTT and websockets
  • It connects to Wifi and MQTT server
  • It sends out heartbeat messages
  • It listens for incoming commands (face management, mode changes etc.). These can be easily sent out by home assistant using your phone, wall panel etc.
  • It starts up a web server that will
    • Stream live video
    • Show logs
  • It has an offline mode in case it can't connect to WiFi/MQTT. If there are no stored faces, it'll automatically try to store first face it sees.

Step 3: Hardware

I found the candy dispenser on Aliexpress but unfortunately its base has been damaged in transport and some patchwork was required.

Good thing about this particular dispenser is that it already has some "smarts" in it - It has hand proximity detection capability which, when hand is detected causes DC motor to run and dispense candy. There is also a white LED that powers on at the same time as the DC motor (this will be handy later on). A pair of IR diodes (transmitter/receiver) is used to detect hand proximity when direct line of sight between these two diodes is interrupted.

I've left the most of above stuff intact and chosen to build on top of that by adding:

  • ESP32-CAM microcontroller to act as a brain that will recognise faces and coordinate dispensing, display messages etc.
  • Relay to allow DC motor to spin only when logic in microcontroller allows it, e.g. when familiar face is recognised.
  • OLED display at the base of dispenser to display messages.
  • Photoresistor (LDR) next to that white LED to help me detect "hand is nearby" event. Alternatively, one could interface directly with dispenser board but I find this "optocoupler-y" isolation of my electronics from existing stuff to be an elegant way of simplifying things and not worry about existing inner circuitry of the dispenser.
  • USB cable as batteries wouldn't last long and I planned to use this indoors anyway.

Steps:

  1. Disassemble the dispenser:
    a) Remove the top candy compartment, black plastic plate and spiral screw with a gear used to push candy out.
    b) Unscrew 6 small screws to release the big plastic cover that holds DC motor and circuit board (let's call this cover A).
    c) Unplug wires from 3 JST connectors (mark which ones goes where as there are two 4-pin connectors and one 2-pin) and remove the cover A.
    d) Unscrew two small screws on a small circuit board and remove that board.
    e) Turn the dispenser upside down, unscrew 3 screws at the bottom and remove the bottom cover.
    f) Unscrew the bigger middle screw.
    g) Dispenser should be fully disassembled now.
  2. Take the cover A with the DC motor on it. Mount the relay there (I used the hot glue). Cut one of the wires leading out the motor and connect loose ends to the relay (COM and NO pins). Drill some holes in the cover to allow wires to pass through to the other side as microcontroller will be placed there.
  3. Mount (hot glue again) the LDR underneath the white LED placed on smaller circuit board (from step 1d).
  4. Make a square hole in the base and mount the OLD display.
  5. Connect all the wires as shown on the diagram. You can solder them but I suggest using dupont connectors wherever you can for easier (dis)assembly. Wiring will be the most pain-sticking step - take your time...
  6. I suggest placing connector for FTDI somewhere handy (I've put it in the base) for easy access and code upload. You'll also need an easy way of connecting GPIO 0 with GND to enable upload mode. Bear in mind that OTA upload is not possible because of the sketch size and ESP32 storage limitations.
  7. Drill a hole on the side of the base to allow 5V power cable to get in (I used USB) as ESP32 is quite power hungry and batteries won't last long especially because we want it to run all the time (so no deep sleep...).
  8. Put everything back together bar the top candy department.
  9. A bunch of wires should be sicking out of the cover A ready to be connected with the ESP32. Before connecting everything, take some time to plan everything, especially where you're going to place camera and ESP32 as camera cable isn't very long. Once you've decided where to put what, cut a thin flat hole in the front plastic to allow camera cable to pass through. Connect the camera with ESP32 and fixate both camera and ESP32 using hot glue. Connect all the wires.
  10. We're ready now to move to the software side....

Step 4: Software

Now that we've put together the dispenser, let's see how to upload the code to ESP-32 microcontroller. I am assuming you'll be using Arduino IDE but feel free to use any alternative (if you know how to set up everything).

First we'll need to set up a board, i.e. ESP-32 microcontroller:

  • Install board esp32 by Espressif Systems (v1.0.4, as newer versions are throwing errors)
  • Create a custom partition (myPartitionForFR.csv)
    • Place myPartitionForFR.csv to your ...AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\tools\partitions\
    • Edit your ...AppData\Local\Arduino15\packages\esp32\hardware\esp32\1.0.4\boards.txt by adding following 3 lines at the bottom of esp32wrover section:
    esp32wrover.menu.PartitionScheme.myPartitionForFR=My FR partitioning - No OTA
esp32wrover.menu.PartitionScheme.myPartitionForFR.build.partitions=myPartitionForFR
esp32wrover.menu.PartitionScheme.myPartitionForFR.upload.maximum_size=2621440
    • Restart Arduino IDE
    • Select "ESP32 Wrover Module" and newly created partition "My FR partitioning - No OTA"

    Now, let's install required libraries:

    • ESP8266 and ESP32 OLED driver for SSD1306 displays by ThingPulse (v4.2.0)
    • PubSubClient by Nick O'Leary (v2.8.0)
    • ArduinoWebSockets by Gil Maimon (v0.5.3)

    And create a new project/sketch:

    • Create a new sketch (File->New) and paste the code from attached CandyDispenser.ino file.
    • Edit relevant config parameters (Wifi, MQTT server if you have one etc.) and save the project. This will create a new folder on your computer.
    • Copy camera_index.h and images.h files to the same project folder. First file holds the HTML/JS code for webserver in Hex format. You can use this site to convert it to human-readable format - recipe should consist of "From Hex" (0x with comma as delimiter) and "Gunzip".
    • Second file has smiley image for OLED.

    Finally, we're ready to upload our sketch to ESP-32:

    • Plug in FTDI serial adapter to the USB on your PC
    • If your FTDI has a voltage jumper, set it to 5V
    • Connect FX, RX, VCC and GND with appropriate connections on the ESP-32 (as shown on the diagram)
    • Put the ESP-32 into upload mode by shorting GPIO 0 and GND (marked as "Upload switch" on the diagram) and restart the ESP-32 by pressing the tact push button switch on the ESP-32
    • Select your board, partition and port (check the attached screenshot)
    • Upload the sketch (CTRL+U)
    • Once you've successfully uploaded the sketch, un-short the GPIO 0 and GND and detach FTDI.

    Let's turn it on!

    If everything went well, you should be able to plug in dispenser into any USB power socket. Once booted it'll automatically enter into "face recording" mode. Using MQTT commands you can subsequently add or delete faces as you like (next step).

Attachments

Step 5: Home Assistant Integration

Hopefully you've managed to put everything together and your candy dispenser is dispensing candy to you (and only you). However, we're quite limited in ways we can interact with our dispenser.

There are different ways you might choose to interact with your smart device: physical buttons/switches, MQTT messages or even using hand gestures. Since I already had MQTT server up and running, that was the most convenient way for me.

MQTT messages can be sent using various tools, e.g. using MQTT.fx tool (now bought by softblade.de) but it isn't very practical having to turn on your PC and type in MQTT messages every time we'd like to interact with our dispenser.

Home Assistant (HA) in combination with MQTT makes everything more user friendly.

In HA, I've created:

  • Binary sensor that listens to heartbeat messages and indicates if dispenser is up and running or not (useful if you're still developing and testing some features).
  • Several buttons that when pressed, send control messages to the dispenser, e.g. store/delete face, turn on/off debugging, change dispensing mode etc.

All that can be either created as smartphone widget or presented as control dashboard on PC or wall panel. I've attached some screenshots and configs to give you an idea how it looks like and how to configure it.

All this is optional and you should choose for yourself the most practical way (if any) to control your dispenser.

Step 6: Video Streaming Server

Video streaming server isn't really required for the finished product but can be useful while developing and testing/debugging. It can be easily turned off by removing Task1 creation command in setup function.

Server is running on Core1 and shouldn't impact face recognition performance that is running on Core0. It can be accessed by any web browser by keying in dispenser's local IP address. Once accessed, it'll display camera view with indicated faces together with log messages.

Many thanks to Robot Zero One as this part has been greatly inspired by one of his projects.

Step 7: Additional Info, Tips and Gotchas:

Hardware:

  • If you're having stability issues, getting "Brownout detector" error etc. try adding capacitor (16V 330uF) across 5V and GND pins. I think it helped me, I think...
  • Even after adding a capacitor I had major WiFi performance issues. Surprisingly "Finger method" did work but that was obviously not a runner. Tried replacing the ESP32-CAM and another microcontroller worked like a charm, so it seems the first one was defective.

Software:

  • There isn't enough memory on ESP32-CAM to support OTA with face recognition libraries. We'd need to have 2 sufficiently large partitions (app0 and app1) to fit the while program. We have enough just for app0, so I suggest you think ahead and expose FTDI ports somewhere handy so that you can upgrade the code without opening up the whole thing every time you want to tweak something.
  • There is a bug that prevents us using analogRead on ADC2 if Wifi is on. There is a workaround that requires us to read some registers and write them back just before invoking analogRead. I'm using it in checkLight function to see if hand-proximity has been triggered. Details here, here and here.
  • ESP32 has two cores! Since face recognition is quite resource hungry task, I've assigned that task to Core 0. Everything else is running on Core 1 (web server runs in a separate task on Core 1).
  • Face recognition won't work well unless there is a good lighting in the room. To improve it, I've lowered some threshold scores (p_threshold, r_threshold, o_threshold). That helped, however we can expect to get a higher number of "false positive" detections. That was acceptable for me as this isn't some high security door/vault lock project, but you should play with the numbers yourself and see what works best for you.
  • I wanted to display smiley on display when primary face is recognised. Had troubles converting images to binary format using other tools suggested on some sites. One that did work for me, also seems to be the simplest one out there.

Other:

  • I used jelly beans but they are not perfect if you would like to dispense the exact number of candy each time as they get sticky and sometimes you get just one and sometimes a bunch of them. I'd suggest something rounder and less sticky (perhaps m&m's) and try not to overfill the container.
  • Put the dispenser next to a good light source.

Step 8: References, Tutorials, Articles...

Here is the list of articles, tutorials and other awesome materials I used while working on my project. Many thanks to all people that spent time to create and document all these, fair play to you!

  1. Robot Zero One - A collection of great ESP32 articles with focus on facial recognition. Here are some that I've found particularly good and useful:
    ->Arduino partitioning
    ->Websockets
    ->Custom html
  2. A brilliant series of YouTube videos made by Shawn Hymel about (free) RTOS that runs on ESP32.
  3. ESP32 microcontroller
    ->Getting started
    ->Troubleshooting
  4. Face recognition libraries
  5. Home assistant and MQTT