RPi Easy Object Detection - La Croix Flavor Detector

3,036

13

19

Introduction: RPi Easy Object Detection - La Croix Flavor Detector

About: Hello, my name is Michael. I am an electrical and computer engineer turned educator. I have a lot of experience in embedded software and hardware design. I am taking my technical experiences and leveraging the…

The Raspberry Pi provides an interesting environment for machine learning. It has a relatively powerful CPU & GPU, is lowcost, supports many of the Linux packages/drivers, and can still be used as a microcontroller to interface with a myriad of different sensors, motors, and other peripherals. Because of this, we have the ability to run some image detection models while controlling hardware to interact with the real world based on what the models detect. The goal of this project is to provide a useful working framework and environment to easily incorporate object detection and machine learning into all of your projects.

We will do this using Teachable Machine. Teachable Machine is an easy, quick, but powerful tool to create machine learning models. It allows for easy data capture to create training data sets and uses state of the art algorithms to train machine learning models right in your browser. It is done in a very intuitive web interface. You can generate image, sound, or pose detection models. In this project, I will provide you with a step-by-step guide for setting up an OpenCV/TensorFlow Python development environment and a Python script framework to easily incorporate Teachable Machine image models into your projects.

The goal of this project is to greatly reduce the barrier to entry to use machine learning. This Instructable should give you the tools you need to make some exciting machine learning projects. I hope to make some more interesting tutorials and demos in the future that use this development environment and framework.

You can follow along and generate your own model or you can use the Teachable Machine model I generated. The model I provided detects what flavor La Croix you have as a fun demonstration of the capabilities.

Github repository for the project: https://github.com/mjdargen/RPi-La-Croix-Flavor-Detector

If you are interested in setting up this environment on your desktop/laptop, here are instructions for Windows/Mac/Linux systems: https://www.instructables.com/id/Easy-Machine-Lear...

Supplies

  • Raspberry Pi (tested on RPi3 Model B and RPi4 Model B)
  • microSD Card (8 GB minimum, 16 GB recommended)
  • HDMI Monitor
  • Mouse/Keyboard
  • Raspberry Pi camera module or USB webcam

Step 1: Generating a Model With Teachable Machine

Setting up the Project

Teachable Machine is a fairly easy-to-use tool with a very intuitive interface. For this project, we will be working with image detection. Go to https://teachablemachine.withgoogle.com/ and click on Get Started. Now select Image Project. This will open up the image model training window.

Creating the Dataset

You will add and name the classes (i.e. objects) you want to train the model to detect. Name the classes well with an intuitive name. The name of the classes is what the later program will call out when that object appears in the frame.

It is a good idea to make a "Background" class. This can help train the model to not attribute details from the background with one of the other classes. If you name this class "Background", the final program, which uses text-to-speech to say the name of the object in the frame, will ignore the background class and not call out "background" every time it is solely the background in frame.

To add image samples to a class, you can either use your webcam to capture images in Teachable Machine or upload images from another source. In order to produce a model, you want a lot of high-quality data. You can see in my example of the "La Croix Flavor Detector Model", I had no less than 600 samples for each class. I used the webcam to quickly capture many different samples. I made sure to capture the object from every angle in different lighting situations with a variety of backgrounds to generate an accurate model.

Training the Model

Once you have set up all of your classes and are happy with your datasets, it is time to train the model! Click the "Train Model" button. In order to train the model, you must leave the tab open in your browser. Training the model can take a while. In this project where I had 7 classes with >600 samples, it took about ~20 minutes to train. Your browser may occasionally complain that the Teachable Machine tab is slowing down your browser. Just acknowledge notification and say it's fine so your browser does not cancel the training (different browsers word this notification differently). Once it's complete, it's time to test out your model!

Previewing the Model

Now it's time to test out your trained model and see how well it does! Go to the Preview pane and turn the input on. Present the various objects to the webcam and see if the model accurately guesses what object is in the frame. Remember, the model cannot detect more than one object unless you made a single class for when two objects are present. If it's not performing well, try providing more photos to the model. If you're happy, it's time to export the model!

Exporting the Model

To export the model, click the "Export Model" button. A new window will pop up. Click the "Tensorflow" tab and select the "Keras" model conversion type. Now click "Download my model". It can take about a minute or so to compress the model and prepare it for download. You should get a pop-up window asking you to save a zip file. Save the file and unzip it. You should see a "keras_model.h5" file and a "labels.txt" file. Hang onto these and we will use them once you have your Python environment set up on your computer!

Step 2: Setting Up Raspberry Pi

If you have not already set up the operating system on your Raspberry Pi, you will want to download "Raspbian Buster with desktop and recommended software" from the Raspberry Pi Foundation's website (https://www.raspberrypi.org/downloads/raspbian/). Any recent version of Raspbian Buster should work fine. This installs the full version with Python 3 along with some programming IDEs. Extract the .img file and write the image to your microSD using an SD card imager program like Rufus, balenaEtcher, Win32DiskImager, or others. The Raspberry Pi Foundation has written up an extensive guide on how to install operating system images onto your Raspberry Pi here.

Once you have finished imaging the SD card, insert it into your Raspberry Pi and power it on. It should do some initial set-up automatically (sizing the screen, expanding the file system, etc.). After the desktop loads, a window will pop up prompting you to follow some set-up steps. You will need to set up your locale, wifi, confirm whether or not the desktop shows a black border around it, allow the Raspberry Pi to update (this can take a while), and then reboot.

After you have successfully updated and rebooted your Raspberry Pi, the desktop should load now without a prompt. Click the Raspberry Pi icon in the top-right corner to show the dropdown menu. Go to "Preferences->Raspberry Pi Configuration". The Raspberry Pi Configuration window will show up. Navigate to the "Interfaces" tab and enable the Camera. A prompt will appear telling you to reboot your Raspberry Pi. Click yes and let you Raspberry Pi reboot.

Once your Raspberry Pi has booted up, open up a terminal to test your camera. Make sure you connect the camera's ribbon cable properly and then type the following command:

raspistill -v -o test.jpg

This will display info about your camera in the terminal and take a photo. It saves the photo as test.jpg and stores it in the directory in which you ran the command. Navigate to that directory in your file browser and open the file to ensure the camera worked properly and captured the photo. More info about using the Raspberry Pi camera is provided at the links below.

https://www.raspberrypi.org/documentation/configur...

https://www.raspberrypi.org/documentation/raspbian...

If you are using a USB webcam, you can follow the steps detailed below.

https://www.raspberrypi.org/documentation/usage/we...

Step 3: Setting Up Raspberry Pi Environment

Python 3 Installation & Set-up

As mentioned before, Python 3 should be installed on the Raspbian image you loaded. To confirm, type "python3 --version". This should return the version number. I have tested this development environment in Python 3.6 and Python 3.7 and everything seemed to work appropriately. However, Python 3.8 did not seem to fully support some of these libraries fully yet. Just make sure you have one of these versions. If you do not have any version installed, you can install it with the following command:

sudo apt-get install python3

Git Project Files

Now you will need to retrieve the installation file, machine learning models, and the demo Python program from my Github repository. As mentioned previously, git should already be installed with your Raspbian image. To confirm, run "git --version" in your terminal and ensure it returns a version. To install, you can fun the following command.

sudo apt-get install git

To download the files required, you can navigate to the repository in your browser and download a zip file or you can use your git client. The link and the git command are described below.

https://github.com/mjdargen/RPi-La-Croix-Flavor-Detector

git clone https://github.com/mjdargen/RPi-La-Croix-Flavor-Detector.git

Step 4: Setting Up Python Environment: OpenCV and TensorFlow

I have written an installation script to simplify the installation process for this development environment. It took me a long, long, long time to figure out all of the specific dependencies to make these packages work on Raspberry Pi. There are some very specific versions required for each of the different packages. Feel free to investigate the installation script or even run the commands one-by-one on your own in the terminal to understand the setup process. After it runs successfully, you should now have OpenCV and Tensorflow to use for any of your machine learning projects!

Raspberry Pi Install

  • Assumes you have Python 3 installed.
  • Assumes that your Python 3 executable is invoked with "python3". If that is not the case, you will need to edit the shell script and replace every instance of the "python3" command with "python".
  • Navigate to the folder of the repository in your terminal.
  • You will need to make the script executable by running the following command: "sudo chmod +x ./rpi_install.sh"
  • Run the shell script with the command: "./rpi_install.sh".
  • Can take ~1.5 hours or more depending upon your system and internet connection.

Using Python Virtual Environments

If the installation script executed successfully, you have now installed all necessary dependencies to run OpenCV and Tensorflow in a Python virtual environment on your machine. The virtual environment is called TMenv and is located in the top-level directory of the cloned repository entitled "Teachable-Machine-Object-Detection".

The Python packages were installed in a virtual environment so as not to disrupt your packages associated with your main installation of Python in case you had other programs that depended upon a specific version of a package.

To use the packages you installed to run the demos, you will need to activate your virtual environment.

source TMenv/bin/activate

Once you have activated your environment, it will show the name of your virtual environment in parenthesis before the prompt in your terminal. Anything you do related to Python at this point will only affect your TMenv virtual environment. You can now run Python scripts in your virtual environment. To exit your virtual environment, just run the command below

deactivate

Step 5: Setting Up Source Code

Now that we have our OpenCV/Tensorflow development environment setup and we have tested it to make sure it works, it's time to move on to running a Teachable Machine model. You can either use the sample model I provided or one that you created and exported.

Once you have successfully exported the model as described in the first step, you will need to unzip the model to extract both the .h5 file and the labels.txt. You will need to update the "model_path" and "labels_path" variables to point to these files in tm_obj_det.py. You will need to determine the width and height of your webcam's video feed in pixels and update the "frameWidth" and "frameHeight" variables. You may also need to mirror the video feed for your webcam depending upon your setup. To do this, uncomment the line "frame = cv2.flip(frame, 1)".

Next, you will need to set your confidence threshold (conf_threshold). This variable is a percentage value of how certain you want the model to be before it labels the image and speaks the prediction. By default, the confidence threshold is 90%.

Finally, if you have any issues with the video showing up properly, you can use the matplotlib implementation. You will need to comment out the "cv2.imshow" and "cv2.waitKey" lines. Then you will need to uncomment "import matplotlib" as well as the plt lines of code towards the end.

That's it, your code is ready to run!

Step 6: Run!

Now your code should be all set up to run. Navigate to the directory, activate your virtual environment, and run the code! It can take about 30 seconds or so to set-up and load the model. After that point, it should load a video feed. The program will label what object it recognizes and will use text-to-speech to say the name of the object.

cd ~/Documents/Teachable-Machine-Object-Detection	# change directory to cloned repo
source TMenv/bin/activate		# activate venv
python tm_obj_det.py			# executes script, press ctrl+c to quit
deactivate				# to exit the virtual environment

Note: When you run the script, you may see a number of different complaints from Python. This is to be expected, your program should still run normally. This is again due to some of the weird dependency issues that you run into on a Raspberry Pi. I had to revert to some older versions of some of the Python packages to get it to work. Because of this, there are some complaints about certain things being deprecated in future versions, etc. I just wanted to make you aware so you didn't think the program was not working.

Note: the Python script will run forever until you hit ctrl+c to close the program.

Step 7: Remix!

These packages installed in your virtual environment and the scripts I provided should hopefully give you a useful framework to develop lots of exciting things. You can now easily incorporate object detection into all of your projects! The Raspberry Pi provides a perfect environment for this since it can easily interface with different sensors, motors, and other peripherals. I hope to continue doing more projects in this space to make some more fun projects that use image detection and leverage this framework.

Here are some project ideas. Feel free to take them and run with them or come up with your own!

  • A program to recognize your friends and greet them by name as they come to your house.
  • A program to detect when you are leaving the house and ask you to present your phone, keys, wallet, etc. to make sure you have everything when you leave the house.
  • Build a sorter that uses a motor to divert objects in a particular direction based on which objects they are.
  • A program that will detect letters in sign language and write these out to a text file.

Step 8: More Projects

Step 9: Source Code

To view the source code, visit this Github repository.

Raspberry Pi Contest 2020

Participated in the
Raspberry Pi Contest 2020

Be the First to Share

    Recommendations

    • Arduino Contest

      Arduino Contest
    • Home Decor Challenge

      Home Decor Challenge
    • Colors of the Rainbow Contest

      Colors of the Rainbow Contest

    19 Comments

    0
    lbw27
    lbw27

    26 days ago

    is it possible to have multiple object detection?

    0
    mjdargen
    mjdargen

    Reply 24 days ago

    If you are asking if you can detect multiple objects in a single program, then yes! In the example I have written up, the program detects 7 different types of objects (the various flavors of La Croix). You will follow the same steps to create and use the model.

    To create your model in Teachable Machine, create a class for each type of object that you want to detect. These objects can be anything! Just make sure you capture enough images so the model can successfully recognize the difference between objects. Always test your model in Teachable Machine before exporting it for use in your Python program to ensure it correctly detects the objects. Thanks for checking out the project!

    0
    lbw27
    lbw27

    Reply 16 days ago

    Sorry, I meant recognizing two different objects at the same time. So if I wanted to place two different types of La-Croix's in the same frame it would be able to call out each flavor.

    0
    mjdargen
    mjdargen

    Reply 15 days ago

    Sorry for the misunderstanding! Recognizing multiple objects in a single frame is rather difficult using the Teachable Machine image models. An easy workaround for that would be to train a class that had both objects in the image. For example, if I was training a model to recognize a banana and an apple, I could create 4 different classes: one for the background, one for the banana, one for the apple, and one for both the banana and apple. That way, it would recognize both in a single frame.

    The better, but slightly more difficult option would be to create multiple models. Running multiple models would require some modification of the code to allow you to toggle back and forth between the models. You would need to create and initialize each model you planned on using in the program. OpenCV captures a single frame at a time to pass to the model for predictions. After capturing the frame, you would then prepare the frame using numpy and converting the color the same as before. Once it was prepared, you would pass the data from the frame to each one of your different models. You would then receive the associated predictions for each model and could combine them accordingly to determine if there are multiple objects present in the single frame. That should give you the functionality you are looking for. Let me know if you have any other questions!

    0
    elena4b
    elena4b

    Question 3 months ago

    I’m a beginner and having a problem. It says “ModuleNotFoundError: No module named ‘cv2’” when I try to run it. How can I get it to run properly?

    0
    mjdargen
    mjdargen

    Answer 3 months ago

    So that means that you have not installed (or it cannot find) OpenCV. Did you run the installation script? Were there any error messages when you executed the installation script? The installation script installs the Python packages inside of a Python virtual environment meaning that you can only see these packages when the virtual environment is activated.

    To make sure OpenCV installed properly in the virtual environment, navigate to the directory where the virtual environment was created and run the command "source TMenv/bin/activate" to activate the virtual environment. Once you've done that, type "pip3 list" to list all Python packages installed. You should see "cvlib" and "opencv-python" in the list.

    If you see those, that means OpenCV was properly installed and you should be able to run the script in the virtual environment. If not, try running the installation script again to create the virtual environment and install all of the dependencies.

    0
    elena4b
    elena4b

    Reply 3 months ago

    I just ran the installation script again and did not see any error messages. When I activate the virtual environment and check pip3 list, I see “cvlib” and “opencv-contrib-python” but not “opencv-python”

    0
    mjdargen
    mjdargen

    Reply 3 months ago

    That is great! That means OpenCV is successfully installed inside your virtual environment. Make sure that you activate your virtual environment and then run the Python script inside of that virtual environment by calling "python3 tm_obj_det.py" from the terminal. You know that you are successfully inside of the virtual environment if you see (TMenv) in your terminal.

    I just tried it from scratch again on my end and everything worked properly. Hopefully, it will work for you.

    0
    elena4b
    elena4b

    Reply 3 months ago

    Yes, it just worked! Thank you for the awesome tutorial.

    0
    mjdargen
    mjdargen

    Reply 3 months ago

    Not a problem! Glad it worked for you!

    0
    niky78
    niky78

    4 months ago

    Thanks for the guide.
    I have completed the whole procedure but when I run the python file it gives me an error on the module.
    No module named 'tensorflow'
    Why? the module is installed
    thank you

    0
    mjdargen
    mjdargen

    Reply 3 months ago

    You are likely getting this error because you are not running the code inside of the virtual environment. The installation script creates a virtual environment called TMenv in which it installs all of the packages. If you do not activate the virtual enviroment each time, you will not be able to access packages such as Tensorflow. I did this so it would not pollute your default Python environment.

    On another note, I just updated the installation script to mitigate some issues with an older version of Tensorflow. I would pull the updates so you can use Tensorflow 2.4.0 and avoid some of the issues that others ran into.

    0
    niky78
    niky78

    Reply 3 months ago

    Hi,
    Perfect, after i run the new script now everything works.
    Now the problem is that the vision is too slow.
    Have I got to use the usb Coral accellerator?
    Thanks

    0
    mjdargen
    mjdargen

    Reply 3 months ago

    Thanks for following up! Glad you were able to get it working! The reaction time of the video is just a factor of how much compute power your Raspberry Pi has. Image detection is pretty computationally intensive. For example, my RPi 3 is quite laggy while my RPi4 with 4GB of RAM is pretty close to real time. A Coral USB Accelerator would definitely help provide more computational power to your Pi and speed up the program. Best of luck!

    0
    niky78
    niky78

    Reply 3 months ago

    Thank you for your answer
    I tried to run the program inside and outside the environment, but it always returns the error.
    I will try the new script on a clean system. I will give feedback

    0
    PhilippeJ15
    PhilippeJ15

    4 months ago

    Hi,

    Thank you for your tutorial. It is very helpful.
    At this date (January, 2021), when you try to open a model generated by teachablemachine, you get some errors due to h5py.
    So, I need to downgrade it to

    pip install h5py==2.10.0

    Next, I get some errors when I try to load the model and I need to uninstall the 1.14.0 version of Tensorflow and install the 2.4.0 version (for RPI 4) :

    pip install https://github.com/bitsy-ai/tensorflow-arm-bin/re...

    Now, it works!

    Philippe

    0
    mjdargen
    mjdargen

    Reply 4 months ago

    Awesome! Glad you were able to get it working. Thanks for providing the update.

    0
    Wahucci
    Wahucci

    1 year ago

    Awesome tutorial! I'm especially thankful for your detailed explanation and installation script as I have struggled long and hard before.
    One thing I noticed when using the Raspberry Pi Camera V2.1:
    With a resolution of 3280x2464, it wouldn't run at all. When changing it to 1920×1080, it was too bright. 1024x640 worked best for me.

    0
    mjdargen
    mjdargen

    Reply 1 year ago

    Thanks so much! I am really happy to hear the installation script worked well for you. It took me a long time to work out all the kinks in the installation steps, but I was determined to get it right so I could easily replicate the environment for myself!

    Glad you were able to get it working with your camera. I was using an older V1.3 which has a different size and aspect ratio. Hope you're able to make some cool stuff now that you have the environment all set up on your Raspberry Pi!