Doorbell With Face Recognition

Motivation

Recently, there has been a wave of robberies in my country which are targeted at elderly people in their own homes. Usually, access is granted by the occupants themselves since the visitors convince them that they are caregivers/nurses. It's just beyond words, how angry and sad these stories make me feel. Home should be your first safe haven and even more so if you are already in a vulnerable position when being outside. With this in mind, I started this project.

General information

The doorbell system is mainly designed for elderly or vision-impaired persons and is pretty straight-forward in its workings. In short, the doorbell switch triggers the camera to obtain footage. Next, faces in the footage are detected and matched with a whitelist and blacklist. The occupant gets clear visual feedback by a clear traffic light display. Hereby, the green, yellow or red light indicate that the person(s) are on the whitelist, unknown to the system or on the blacklist respectively. If the yellow or red light is triggered, the photo is sent by a Telegram bot to inform/warn a relative or caretaker.

Level of expertise

The project was set up for enthusiasts that are particularly curious about using computer vision and artificial intelligence. This instructable is written for an audience of beginners, so don't worry if you have no experience! Furthermore, the project can be interesting for more experienced makers as well since the pipeline is organized in a way that you can expand it with your own computer vision and face recognition ideas without much hassle.

Step 1: Materials

Product list with minimum requirements:

Product	Link	Comment
Raspberry Pi 3b	RPi	Link shows RPi 4 since it has way better performance and almost equal price as the RPi 3b.
Micro SD	Amazon	An micro SD card of 16 GB or larger will do the job. But the 16 GB cards on Amazon are now pretty much the same price as the 32 GB cards.
Raspberry Pi camera	Amazon	Camera v1 is cheaper, but v2 is better and will be longer supported.
15 pin FPC flex cable	Amazon	The length actually depends on the circumstances of doing this project. If you just want to build a prototype, the original flex cable will do the job.
Power Supply 5v micro usb	Adafruit	This one never let met down! Great quality. (Not on the photo)
Arcade buttons with built-in LED	Amazon	Choose the size you want, but the CAD design is based upon 60mm buttons
Resistors	Amazon	You just need a couple of 1k and 100 ohm resistors. Regular 1/4W's are fine.
Capacitors 0.1 uF	Amazon	Three capacitors are needed. (Not on the photo)
Jumper Wires / Ribbon Cable	AmazonAmazon	If you want to save yourself some bucks, you could also use an old floppy drive ribbon cable (see photo).
Shrinking tube / Electrical tape	Amazon Amazon

Tools that are needed:

Tool	Essential?	Comment
Soldering iron	Yes
Multimeter	Yes
Wire stripper	Yes	Or you can use a knife/scissors.
Laser Cutter	No
3D printer	No
Clamps	No	Useful for keeping the box together in the testing phase.

Remarks:

• To increase accessibility of the project, I decided to develop it using a Raspberry Pi 3b. While it increases the accessibility, it decreases the capabilities of the application since RPi's are not that fast. If you are looking for a single board computer that is faster, you might want to take a look at the NVIDIA Jetson Nano.

Step 2: Wiring

The schematic diagram is most informative for this step and is quite self explanatory. In case you are new to electronics, you can use the legend image. The value of the component (if applies) is stated in the schematic diagram. The photos may help to see how I build the circuit. Basically, I connected all components as close to the arcade button as possible which results in a clear overview of what's going on.

Remarks:

• I really like to use ribbon cable connectors, since they are way sturdier than using single jumper wires.
• As suggested, I have used a scavenged ribbon cable from an old computer. This is a bit tricky though since you'll have to manually assert the configuration of the cable. In this project for example, I found out that some holes were connected with each other (probably used as ground for the original application). Therefore, I had to get a different cable later as you can see on the pictures.

Step 3: Building the Casing

Camera casing

Many casings for the picamera can be freely downloaded from the internet. So, I choose not to reinvent the wheel and pick a basic but nice casing from the internet: thingiverse.com - Raspberry pi camera case/enclosure. (Shout-out to the designer VGer.)

Traffic light casing

For the traffic light casing, I designed a little box in Autodesk Fusion 360 (which is freely downloadable, see Remarks) that fits all the hardware. In the attachment, you can find the file that I have send to my local laser cutting company. Hereby, the design is based on a 6mm plate thickness. However, if you want to adjust things, you can access all kinds of file formats by using this link. As shown in the pictures, you could also use a cardboard box if you don't have access to a laser cutter. I used the cardboard box on the picture for prototyping and it works like a charm.

The assembly is pretty straight forward:

1. Mount the Arcade switches.
2. Make sure to keep the wires for the doorbell free.
3. Connect the ribbon cable to the RPi.
4. Screw the RPi onto the bottom panel.
5. Connect the doorbell wires to a wire connector and mount it to the bottom panel also.
6. Connect the Picamera to the RPi.
7. Drill a hole in one of the side panels for the doorbell switching wire and the RPi power wire.

The wire connector is used as a mounting point for the doorbell switch wires, so that it can be fixed to an existing doorbell later on. Everything is now in place and can be glued together. However, you might first want to finish the next steps, to make sure everything works the way it is supposed to be.

Remarks:

• Autodesk Fusion 360 is freely available for hobbyists! If you want to get your copy, visit this link: autodesk.com - Fusion 360 For Hobbyists. There are some terms, so make sure to read and apply them. It was my first project with Fusion 360 and I don't have a lot experience in using CAD software, but I must say that I really like the software and all the additional tools coming with Fusion 360.

Step 4: Configuring the Camera

It is assumed that you have Raspbian installed and that it runs in GUI mode. If you don't have Raspbian installed yet, you can follow this article: raspberrypi.org - Installing operating system images. If you boot Raspbian, you should see a desktop as shown in the pictures.

Let's configure the camera on the RPi and see if it works! The method described here is directly from the raspberrypi.org - Documentation. First, let's update to the latest packages (including camera firmware) by executing the following commands in a terminal window (see pictures):

sudo apt update
sudo apt full-upgrade

Next, the camera has to be enabled using the following command:

sudo raspi-config

In the menu, go to 5. Interfacing Options -> P1 Camera. Choose to enable the camera and reboot the RPi by executing:

reboot

The camera should now be configured properly. It can be tested by opening a terminal window and execute:

raspistill -v -o /home/pi/test.jpg

The image is saved to: /home/pi.

Step 5: Setting Up Docker

To avoid the dependency and installation errors, I decided to build a custom Docker image for this project (see wikipedia.org - Docker). If you never used or heard of Docker, no worries, I will explain step by step how to use it in this project. In fact, it's super easy! In case you want to run this project on a local install (instead of in a Docker container), I will give you some tips. But it is highly recommended to use the Docker image. After all, I build it to make it easy for you to run this project!

What is Docker?

Note: this part provides some background information about Docker, which can be skipped if you just want to run the code.

This project is the first time I used Docker and it is simply awesome! Maybe you have heard about virtualenv or Anaconda for Python? Well, Docker is pretty similar in the sense that you can easily manage package versions and run different Python versions on one host system by using a different environment (or container as it is called in Docker). But, compared to virtualenv and Anaconda, Docker is way more powerful since it is not limited to contain Python packages only. Indeed, in a Docker container, you can install and manage also the packages of a desired operating system. For example, consider a website you want to migrate that runs a Python web framework (e.g. Django) with a database (e.g MySQL). Without a Docker container, you would have to install all packages on the new server, a process which is very prone to errors and bugs. On the other hand, when your website has been build in Docker, migrating is basically as easy as moving the image file/files to the new server and run it/them. As you can imagine, Docker is very useful for projects on Instructables as well ;)! If you want to know more about Docker, check out their website: docker.org - Docker: Enterprise Container Platform. Now, let's get up and running with Docker!

Installing Docker

Install Docker by executing:

curl -sSL https://get.docker.com | sh

Next, the user is added to the usergroup 'docker', which provides the rights to run Docker. This is done by:

sudo usermod -aG docker $USER

Now, you should be able to run Docker. This can be validated by running the hello-world image:

docker run hello-world

Lastly, let's pull the Docker image that contains all dependencies needed to run the doorbell Python scripts. This process might take a while since the image is quite large (~1.5 GB). Execute:

docker pull erientes/doorbell

Note: the Dockerfile can be found on the doorbell repository on Github.

Now, everything is ready to run the doorbell scripts, which will be discussed in the next step.

Local installation

Again, I would highly recommend to use the Docker image instead of a local install. But to make this tutorial complete, I will now describe some of the steps I took for the local install.

To be able to run the code, the python version should be >= 3.5 (I used python 3.5.3) and the following packages need to be installed:

• face_recognition
• picamera
• numpy
• Pillow
• python-telegram-bot
• RPi.GPIO

This link is quite helpful: Github - Install dlib and face_recognition on a Raspberry Pi. However, there are some caveats here: 1) Pillow needs at least Python 3.5, which won't be installed following this method. 2) Also, not all packages that are needed in the doorbell project will be installed by following this method. However, you should be able to install it by simply using pip3.

Step 6: Running the Doorbell Scripts

Get the scripts

The scripts can be manually downloaded from: github.com - Erientes/doorbell. Or if you have Git installed, execute:

git clone https://github.com/Erientes/doorbell

Create aliases

Now, to make our lives a bit easier, let's create some aliases to run the scripts. Execute:

leafpad ~/.bashrc

Add the following lines and save the file:

alias doorbell_run='docker run --privileged -v /home/pi/doorbell:/doorbell -w /doorbell -it erientes/doorbell python $1'

alias doorbell_login='docker run --privileged -v /home/pi/doorbell:/doorbell -w /doorbell -it erientes/doorbell bash'

Test scripts

To test if everything is installed right, open a new terminal and execute:

doorbell_run examples/0_test_installation.py

The result should simply be a message in the terminal window that says 'Doorbell installation ended successfully!'. To test if the camera can be accessed by the Docker container, run:

doorbell_run examples/1_test_camera.py

By running 1_test_camera.py a photo will be taken and saved as 'test.jpg', which can be found in /home/pi/doorbell. Lastly, the LED drivers can be tested by executing:

doorbell_run examples/2_test_voicehat_drivers.py

When this script is running, the LED in the arcade switch should respond when the button is pressed.

Running Doorbell scripts

To run the Doorbell scripts, first the Telegram bot credentials need to be obtained. Install Telegram on your phone and go to telegram.me - Botfather. Start a conversation and enter:

/newbot

Fill in a name and an username for the bot. After that, you will be provided the access token. Copy that value to the file 'credentials_telegram_template.py' in /home/pi/doorbell and save it to a new file named 'credentials_telegram.py'. Lastly, start a conversation with the bot you just created by clicking on the link that Botfather provides you.

Finally, let's run the Doorbell with Face Recognition:

doorbell_run main.py

Remarks:

• If you want to know more about how the code works, check out the comments in the scripts themselves. If you have a question about the code, please contact me via Github.

Step 7: Using the Doorbell

Let's run the doorbell script by executing:

doorbell_run main.py

After loading the packages, the scripts goes idle. There are basically 2 things that can happen:

1. Someone rings the doorbell.
2. Someone is added to the whitelist.

Someone rings the doorbell

In this case, the script will start taking photos until it shoots a photo in which a face is detected. After detection, some methods from python package 'face_recognition' are called to calculate a 128 encoding of the face. Next, the obtained encoding is compared to the encodings in whitelist.csv and blacklist.csv. The possible outcomes result in the following response:

In whitelist?	In blacklist?	Response
Yes	No	Green light turns on.
Yes	Yes	Yellow light turns on. The doorbell camera sends photos to Telegram bot with orange icon. This state can happen if someone was added to both lists. For example, when someone was welcome at first, but was blacklisted later.
No	No	Yellow light turns on. The doorbell camera sends photos to Telegram bot with orange icon.
No	Yes	Red light turns on. The doorbell camera sends photos to Telegram bot with red icon.

Someone is added to the whitelist

To add someone to the whitelist, press the yellow button of the traffic light when the doorbell is in idle state. First, the yellow light will turn on. If the green light blinks 3 times, the person's face is successfully added to the whitelist. If the green light doesn't blink 3 times, the attempt was not successful. In that case, press the yellow button again. You can easily verify if it was successful by ringing the doorbell and checking if the green light is passed.

How to add someone to the blacklist?

Obviously, people with bad intentions won't pass by to give us a photo of their face. So instead, you can add images of notorious people that (for example) the police has published to the folder img/blacklist. Every hour, this folder is checked for new images. If there is a new image, the face encoding is calculated and added to blacklist.csv. The image is then renamed and moved to the folder /img/blacklist/encoded.

Remarks:

• Operating the scripts by login into the RPi provides way more control and information, but the basic control and information can be obtained by solely using the traffic light display.
• Face recognition is implemented by using python package 'face_recognition'. This package is based on Dlib that contains a state-of-the-art face recognition algorithm, which performs an accuracy of 99.38% on the Labeled Faces in the Wild benchmark (source: dlib.net - High Quality Face Recognition with Deep Metric Learning).