Introduction: RPi Easy Object Detection - La Croix Flavor Detector

By https://dargen.io/
About: Hello, my name is Michael. I am an electrical and computer engineer turned educator. I have a lot of experience in embedded software and hardware design. I am taking my technical experiences and leveraging the…

The Raspberry Pi provides an interesting environment for machine learning. It has a relatively powerful CPU & GPU, is lowcost, supports many of the Linux packages/drivers, and can still be used as a microcontroller to interface with a myriad of different sensors, motors, and other peripherals. Because of this, we have the ability to run some image detection models while controlling hardware to interact with the real world based on what the models detect. The goal of this project is to provide a useful working framework and environment to easily incorporate object detection and machine learning into all of your projects.

We will do this using Teachable Machine. Teachable Machine is an easy, quick, but powerful tool to create machine learning models. It allows for easy data capture to create training data sets and uses state of the art algorithms to train machine learning models right in your browser. It is done in a very intuitive web interface. You can generate image, sound, or pose detection models. In this project, I will provide you with a step-by-step guide for setting up an OpenCV/TensorFlow Python development environment and a Python script framework to easily incorporate Teachable Machine image models into your projects.

The goal of this project is to greatly reduce the barrier to entry to use machine learning. This Instructable should give you the tools you need to make some exciting machine learning projects. I hope to make some more interesting tutorials and demos in the future that use this development environment and framework.

You can follow along and generate your own model or you can use the Teachable Machine model I generated. The model I provided detects what flavor La Croix you have as a fun demonstration of the capabilities.

Github repository for the project: https://github.com/mjdargen/RPi-La-Croix-Flavor-Detector

If you are interested in setting up this environment on your desktop/laptop, here are instructions for Windows/Mac/Linux systems: https://www.instructables.com/id/Easy-Machine-Lear...

Supplies

  • Raspberry Pi (tested on RPi3 Model B and RPi4 Model B)
  • microSD Card (8 GB minimum, 16 GB recommended)
  • HDMI Monitor
  • Mouse/Keyboard
  • Raspberry Pi camera module or USB webcam

Step 1: Generating a Model With Teachable Machine

Setting up the Project

Teachable Machine is a fairly easy-to-use tool with a very intuitive interface. For this project, we will be working with image detection. Go to https://teachablemachine.withgoogle.com/ and click on Get Started. Now select Image Project. This will open up the image model training window.

Creating the Dataset

You will add and name the classes (i.e. objects) you want to train the model to detect. Name the classes well with an intuitive name. The name of the classes is what the later program will call out when that object appears in the frame.

It is a good idea to make a "Background" class. This can help train the model to not attribute details from the background with one of the other classes. If you name this class "Background", the final program, which uses text-to-speech to say the name of the object in the frame, will ignore the background class and not call out "background" every time it is solely the background in frame.

To add image samples to a class, you can either use your webcam to capture images in Teachable Machine or upload images from another source. In order to produce a model, you want a lot of high-quality data. You can see in my example of the "La Croix Flavor Detector Model", I had no less than 600 samples for each class. I used the webcam to quickly capture many different samples. I made sure to capture the object from every angle in different lighting situations with a variety of backgrounds to generate an accurate model.

Training the Model

Once you have set up all of your classes and are happy with your datasets, it is time to train the model! Click the "Train Model" button. In order to train the model, you must leave the tab open in your browser. Training the model can take a while. In this project where I had 7 classes with >600 samples, it took about ~20 minutes to train. Your browser may occasionally complain that the Teachable Machine tab is slowing down your browser. Just acknowledge notification and say it's fine so your browser does not cancel the training (different browsers word this notification differently). Once it's complete, it's time to test out your model!

Previewing the Model

Now it's time to test out your trained model and see how well it does! Go to the Preview pane and turn the input on. Present the various objects to the webcam and see if the model accurately guesses what object is in the frame. Remember, the model cannot detect more than one object unless you made a single class for when two objects are present. If it's not performing well, try providing more photos to the model. If you're happy, it's time to export the model!

Exporting the Model

To export the model, click the "Export Model" button. A new window will pop up. Click the "Tensorflow" tab and select the "Keras" model conversion type. Now click "Download my model". It can take about a minute or so to compress the model and prepare it for download. You should get a pop-up window asking you to save a zip file. Save the file and unzip it. You should see a "keras_model.h5" file and a "labels.txt" file. Hang onto these and we will use them once you have your Python environment set up on your computer!

Step 2: Setting Up Raspberry Pi

If you have not already set up the operating system on your Raspberry Pi, you will want to download "Raspbian Buster with desktop and recommended software" from the Raspberry Pi Foundation's website (https://www.raspberrypi.org/downloads/raspbian/). Any recent version of Raspbian Buster should work fine. This installs the full version with Python 3 along with some programming IDEs. Extract the .img file and write the image to your microSD using an SD card imager program like Rufus, balenaEtcher, Win32DiskImager, or others. The Raspberry Pi Foundation has written up an extensive guide on how to install operating system images onto your Raspberry Pi here.

Once you have finished imaging the SD card, insert it into your Raspberry Pi and power it on. It should do some initial set-up automatically (sizing the screen, expanding the file system, etc.). After the desktop loads, a window will pop up prompting you to follow some set-up steps. You will need to set up your locale, wifi, confirm whether or not the desktop shows a black border around it, allow the Raspberry Pi to update (this can take a while), and then reboot.

After you have successfully updated and rebooted your Raspberry Pi, the desktop should load now without a prompt. Click the Raspberry Pi icon in the top-right corner to show the dropdown menu. Go to "Preferences->Raspberry Pi Configuration". The Raspberry Pi Configuration window will show up. Navigate to the "Interfaces" tab and enable the Camera. A prompt will appear telling you to reboot your Raspberry Pi. Click yes and let you Raspberry Pi reboot.

Once your Raspberry Pi has booted up, open up a terminal to test your camera. Make sure you connect the camera's ribbon cable properly and then type the following command:

raspistill -v -o test.jpg

This will display info about your camera in the terminal and take a photo. It saves the photo as test.jpg and stores it in the directory in which you ran the command. Navigate to that directory in your file browser and open the file to ensure the camera worked properly and captured the photo. More info about using the Raspberry Pi camera is provided at the links below.

https://www.raspberrypi.org/documentation/configur...

https://www.raspberrypi.org/documentation/raspbian...

If you are using a USB webcam, you can follow the steps detailed below.

https://www.raspberrypi.org/documentation/usage/we...

Step 3: Setting Up Raspberry Pi Environment

Python 3 Installation & Set-up

As mentioned before, Python 3 should be installed on the Raspbian image you loaded. To confirm, type "python3 --version". This should return the version number. I have tested this development environment in Python 3.6 and Python 3.7 and everything seemed to work appropriately. However, Python 3.8 did not seem to fully support some of these libraries fully yet. Just make sure you have one of these versions. If you do not have any version installed, you can install it with the following command:

sudo apt-get install python3

Git Project Files

Now you will need to retrieve the installation file, machine learning models, and the demo Python program from my Github repository. As mentioned previously, git should already be installed with your Raspbian image. To confirm, run "git --version" in your terminal and ensure it returns a version. To install, you can fun the following command.

sudo apt-get install git

To download the files required, you can navigate to the repository in your browser and download a zip file or you can use your git client. The link and the git command are described below.

https://github.com/mjdargen/RPi-La-Croix-Flavor-Detector

git clone https://github.com/mjdargen/RPi-La-Croix-Flavor-Detector.git

Step 4: Setting Up Python Environment: OpenCV and TensorFlow

I have written an installation script to simplify the installation process for this development environment. It took me a long, long, long time to figure out all of the specific dependencies to make these packages work on Raspberry Pi. There are some very specific versions required for each of the different packages. Feel free to investigate the installation script or even run the commands one-by-one on your own in the terminal to understand the setup process. After it runs successfully, you should now have OpenCV and Tensorflow to use for any of your machine learning projects!

Raspberry Pi Install

  • Assumes you have Python 3 installed.
  • Assumes that your Python 3 executable is invoked with "python3". If that is not the case, you will need to edit the shell script and replace every instance of the "python3" command with "python".
  • Navigate to the folder of the repository in your terminal.
  • You will need to make the script executable by running the following command: "sudo chmod +x ./rpi_install.sh"
  • Run the shell script with the command: "./rpi_install.sh".
  • Can take ~1.5 hours or more depending upon your system and internet connection.

Using Python Virtual Environments

If the installation script executed successfully, you have now installed all necessary dependencies to run OpenCV and Tensorflow in a Python virtual environment on your machine. The virtual environment is called TMenv and is located in the top-level directory of the cloned repository entitled "Teachable-Machine-Object-Detection".

The Python packages were installed in a virtual environment so as not to disrupt your packages associated with your main installation of Python in case you had other programs that depended upon a specific version of a package.

To use the packages you installed to run the demos, you will need to activate your virtual environment.

source TMenv/bin/activate

Once you have activated your environment, it will show the name of your virtual environment in parenthesis before the prompt in your terminal. Anything you do related to Python at this point will only affect your TMenv virtual environment. You can now run Python scripts in your virtual environment. To exit your virtual environment, just run the command below

deactivate

Step 5: Setting Up Source Code

Now that we have our OpenCV/Tensorflow development environment setup and we have tested it to make sure it works, it's time to move on to running a Teachable Machine model. You can either use the sample model I provided or one that you created and exported.

Once you have successfully exported the model as described in the first step, you will need to unzip the model to extract both the .h5 file and the labels.txt. You will need to update the "model_path" and "labels_path" variables to point to these files in tm_obj_det.py. You will need to determine the width and height of your webcam's video feed in pixels and update the "frameWidth" and "frameHeight" variables. You may also need to mirror the video feed for your webcam depending upon your setup. To do this, uncomment the line "frame = cv2.flip(frame, 1)".

Next, you will need to set your confidence threshold (conf_threshold). This variable is a percentage value of how certain you want the model to be before it labels the image and speaks the prediction. By default, the confidence threshold is 90%.

Finally, if you have any issues with the video showing up properly, you can use the matplotlib implementation. You will need to comment out the "cv2.imshow" and "cv2.waitKey" lines. Then you will need to uncomment "import matplotlib" as well as the plt lines of code towards the end.

That's it, your code is ready to run!

Step 6: Run!

Now your code should be all set up to run. Navigate to the directory, activate your virtual environment, and run the code! It can take about 30 seconds or so to set-up and load the model. After that point, it should load a video feed. The program will label what object it recognizes and will use text-to-speech to say the name of the object.

cd ~/Documents/Teachable-Machine-Object-Detection	# change directory to cloned repo
source TMenv/bin/activate		# activate venv
python tm_obj_det.py			# executes script, press ctrl+c to quit
deactivate				# to exit the virtual environment

Note: When you run the script, you may see a number of different complaints from Python. This is to be expected, your program should still run normally. This is again due to some of the weird dependency issues that you run into on a Raspberry Pi. I had to revert to some older versions of some of the Python packages to get it to work. Because of this, there are some complaints about certain things being deprecated in future versions, etc. I just wanted to make you aware so you didn't think the program was not working.

Note: the Python script will run forever until you hit ctrl+c to close the program.

Step 7: Remix!

These packages installed in your virtual environment and the scripts I provided should hopefully give you a useful framework to develop lots of exciting things. You can now easily incorporate object detection into all of your projects! The Raspberry Pi provides a perfect environment for this since it can easily interface with different sensors, motors, and other peripherals. I hope to continue doing more projects in this space to make some more fun projects that use image detection and leverage this framework.

Here are some project ideas. Feel free to take them and run with them or come up with your own!

  • A program to recognize your friends and greet them by name as they come to your house.
  • A program to detect when you are leaving the house and ask you to present your phone, keys, wallet, etc. to make sure you have everything when you leave the house.
  • Build a sorter that uses a motor to divert objects in a particular direction based on which objects they are.
  • A program that will detect letters in sign language and write these out to a text file.

Step 8: More Projects

For more projects, visit these links:

To support me, go here: https://www.buymeacoffee.com/mjdargen

Raspberry Pi Contest 2020

Participated in the
Raspberry Pi Contest 2020