Image Recognition With K210 Boards and Arduino IDE/Micropython

19,970

52

58

Introduction: Image Recognition With K210 Boards and Arduino IDE/Micropython

About: https://www.youtube.com/c/hardwareai My channel about robotics with ROS and machine learning!

I already wrote one article on how to run OpenMV demos on Sipeed Maix Bit and also did a video of object detection demo with this board. One of the many questions people have asked is - how can I recognize an object that the neural network is not trained for? In other words how to make your own image classifier and run it with hardware acceleration.

This is an understandable question, since for your project you probably don't need to recognize some generic objects, like cats and dogs and airplanes. You want to recognize something specific, for example, a breed of the dog for that automatic pet door, or a plant species for sorting, or any other exiting applications you can think about!

I got you! In this article I will teach you how to create your own custom image classifier with transfer learning in Keras, convert the trained model to .kmodel format and run it on Sipeed board (can be any board, Bit/Dock or Go) using Micropython or Arduino IDE. And only your imagination will be the limit to tasks you can do with this knowledge.

UPDATE MAY 2020: Seeing how my article and video on Image Recognition with K210 boards are still very popular and among top results on YouTube and Google, I decided to update the article to include the information about aXeleRate, Keras-based framework for AI on the Edge I develop.

aXeleRate, essentially, is based off the collection of scripts I used for training image recognition/object detection models - combined into a single framework and optimized for workflow on Google Colab. It is more convenient to use and more up to date.

For the old version of the article, you can still see it on steemit.com.

Step 1: CNN and Transfer Learning: Some Theory

Convolutional Neural Networks or CNN is a class of deep neural networks, most commonly applied to analyzing visual imagery. There is a lot of literature on the internet on the topic and I'll give some links in the last part of the article. In short, you can think of CNN as a series of filters, applied to the image, each filter looking for a specific feature in the image - on the lower convolutional layers the features are usually lines and simple shapes and on the higher layers the features can be more specific, e.g. body parts, specific textures, parts of animals or plants, etc. A presence of certain set of features can give us a clue to what the object in the image might be. Whiskers, two eyes and a black nose? Must be cat! Green leaves, a tree trunk? Looks like a tree!

I hope you get the idea about the working principle of CNN now. Normally a deep neural network needs thousands of images and hours of training time(depends on the hardware you are using for training) to "develop" filters that are useful for recognizing the types of objects you want. But there is a shortcut.

A model trained to recognize a lot of different common objects(cats, dogs, house appliances, transport, etc) already has a lot of those useful filters "developed", so we don't need it to learn recognizing the basic shapes and parts of the objects again. We can just re-train the last few layers of the network to recognize specific classes of objects, that are important for us. This is called "transfer learning". You need significantly much less training data and compute time with transfer learning, since you are only training last few layers of the network, composed maybe of few hundred neurons.

Sounds awesome, right? Let's see how to implement it.

Step 2: Prepare Your Environment

To are two ways to use aXeleRate: running locally on Ubuntu machine or in Google Colab. For running in Google Colab, have a look at this example:

Image classification Colab Notebook

Training your model locally and exporting it to be used with hardware acceleration is also much easier now.

My working environment is Ubuntu 16.04, 64bit. You can use Virtual machine to run Ubuntu image since we will not use GPU for training. With some modifications you can also run the training script on Windows, but for model conversion you will need to use Linux system. So, preferable environment for you to execute this tutorial is Ubuntu 16.04, running natively or in virtual machine.

Let's start by installing Miniconda, which is environment manager for Python. We will create isolated environment, so we won't accidentally change anything in your system Python environment.

Download the installer here

After installation is complete, create a new environment:

conda create -n ml python=3.7

Let's activate the new environment

conda activate ml

A prefix before your bash shell will appear with the name of the environment, indicating that you work now in that environment.

Step 3: Install AXeleRate and Run Tests

Install aXeleRate on your local machine with

pip install git+https://github.com/AIWintermuteAI/aXeleRate

To download examples run:

git clone https://github.com/AIWintermuteAI/aXeleRate

You can run quick tests with tests_training.py in aXeleRate folder. It will run training and inference for each model type, save and convert trained models. Since it is only training for 5 epochs and dataset is very small, you will not be able to get useful models, but this script is only meant for checking for absence of errors.

Step 4: Re-train the Model, Convert Keras Model to .kmodel

For this toy example we will be training the model to recognize Santa Claus and Arduino Uno. Obviously you can choose other classes. Download the dataset from here. Create a copy of classifier.json file in config folder, then change it accordingly, similar to config file in the screenshot - make sure the path to training and validation folders is correct!

Run the following command from aXeleRate folder:

python axelerate/train.py - c configs/santa_uno.json

The training will start. If the validation accuracy(our validation metric) is not improving for 20 epochs, the training will stop prematurely. Every time validation accuracy improves, model is saved in the project folder. After training is over, aXeleRate automatically converts the best model to specified formats - you can choose, "tflite", "k210" or "edgetpu" as of now.

Step 5: Run the Model on Sipeed Maix Bit

There are two ways to run the model you have now on Sipeed Maix hardware: micropython firmware and Arduino IDE. Micropython hardware is easier to use, but it occupies significant portion of available memory, so there is less space left for the model. Arduino IDE is basically C code, which is much more efficient and has smaller memory footprint. My model is just 1.9Mb, so both options work for it. You can use models as large as 2.9 Mb with Micropython, for anything larger you need to consider using Arduino IDE.

Download OpenMV IDE from here and minimal micropython firmware from here.

Burn the firmware with kflash_gui tool. You can also choose to burn the trained model to flash too, as shown in the screenshot. Or copy it to SD card(in that case copy .kmodel to the root of an SD card and insert SD card into Sipeed Maix Bit)

Open OpenMV IDE and press the connect button. Open santa_uno.py script from example_scripts folder and press Start button. You should be seeing a live stream from camera and if you open Serial Terminal you will the top image recognition result with the confidence score!

For using with Arduino IDE, first you need to follow the procedure for adding Sipeed boards to Arduino IDE, which is documented here. Your Arduino IDE version needs to be at least 1.8.12. After you added the boards, open the mobilenet_v1_transfer_learning.ino sketch and upload it to Sipeed Maix Bit. Change the name of the model on SD card to "model" (or make a copy with this name). You can change the label names in names.cpp. It will show the live camera stream on the Sipeed Maix screen along with the top image recognition result.

Step 6: Conclusions

Here are some more materials to read on the topic of CNNs and transfer learning:

Transfer Learning using Mobilenet and Keras A great explanation of Transfer learning, this tutorial uses a modified version of the code from that article.

Cats and dogs and convolutional neural networks Explains basics behind CNNs and visualizes some of the filters. With cats!

Train, Convert, Run MobileNet on Sipeed MaixPy and MaixDuino! A tutorial from the Sipeed team on how to train Mobilenet 1000 classes from scratch(no transfer learning). You can download their pre-trained model and try it out!

Hope you can use the knowledge you have now to build some awesome projects with machine vision! You can buy Sipeed boards here, they are among the cheapest options available for ML on embedded systems.

Add me on LinkedIn if you have any question and subscribe to my YouTube channel to get notified about more interesting projects involving machine learning and robotics.

1 Person Made This Project!

Recommendations

  • Audio Challenge 2020

    Audio Challenge 2020
  • Maps Challenge

    Maps Challenge
  • Backyard Contest

    Backyard Contest

58 Discussions

0
jpelletier
jpelletier

8 weeks ago

Fantastic tutorial! Amazing platform, but in Arduino, the board manager URL doesn't work anymore.

0
DmitryM8
DmitryM8

Reply 5 weeks ago

I checked it just now, it does work, but Arduino IDE version needs to be at least 1.8.12 :) it didn't work with 1.8.9

0
amtcChris
amtcChris

7 weeks ago

Nice tutorial! Keep up the good work on aXeleRate!

Note on how to get it working on fresh installations (in my case Debian under WSL):
When using the tests_training.py of aXeleRate, there will be an error caused by sklearn.utils.linear_assignment_ being deprecated.
Use the following command to create the conda environment with a suitable version of scikit-learn:
conda create -n ml python=3.7 "scikit-learn<0.23"

0
DmitryM8
DmitryM8

Reply 5 weeks ago

Thanks for the feedback! I need to correct that

0
thuffam
thuffam

Tip 3 months ago

Thanks for this article - very useful!
I had a couple of errors running test.py - but I've found the solution - hopefully it'll help someone else...
Error 1: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
Solution: This requires an older version of numpy:
conda install "numpy<1.17.0"
Error 2: AttributeError: module 'tensorflow.python.keras.backend' has no attribute 'get_graph'
Solution: This requires an older version of kerass:
conda install keras=2.2.4

HTH
Tim

0
DmitryM8
DmitryM8

Reply 2 months ago

Thank you for noticing this! I updated the article recently to use my new Keras-based framework, aXeleRate to train the models.

0
petia.d66
petia.d66

2 months ago

Just out of curiosity, is there a reason while mobile nets v1 is used rather than v2? :)

0
DmitryM8
DmitryM8

Reply 2 months ago

Mobilenet v2 has more complicated structure, including skip connections between blocks. At the time of original article nncase converter didn't support converting models with skip connections :)

1
xharkx
xharkx

6 months ago

Hello. Thank you for your tutorial.
I spend a day and succes build .kmodel.
but, I got error on mobilenet.py

<line>
fmap = kpu.forward(task, img)
<error>
ValueError:[MAIXPY]kpu: check img format err!

I also use your .kmodel,but still get same error.
https://github.com/AIWintermuteAI/transfer_learning_sipeed/tree/master/model_labels/model.kmodel

Any ideas why?

I also notice,my .kmodel file size always 1.87 MB (1,964,232 bytes).
even I change images.Is this normal?

Pardon my broken English.
Thank you.

0
DmitryM8
DmitryM8

Reply 4 months ago

Model should be the same size, since the number of connections is the same, regardless of the images you're trying to classify (that is of course if you don't change model architecture).
This is weird... Have you made any changes to mobilent.py? What board and firmware are you using?

0
xharkx
xharkx

Reply 4 months ago

Thanks for the reply.
I think I didn't change mobilent.py. I use Sipeed Maix M1 Dock board.
I spent many days and I gave up. I deleted my Virtual machine environment.
But you reply to me,so I will try again from start.

0
kalyanas2
kalyanas2

6 months ago

Hello .when converting from tflite to kmodel i am getting this error
./tflite2kmodel.sh: line 5: ./ncc/ncc: No such file or directory

please help me

0
MarlonH11
MarlonH11

Reply 5 months ago

I had the same problem.
You need to go to the ncc/ directory and and extract the archiv with tar -xf archivname.tar.xz

then you should be able to convert into kmodel.

I hope its helpful

0
avdbrink
avdbrink

8 months ago

Hi Dimtri,

Thanks for this wonderful tutorial. First one for me on this subject and it's very helpful so far.

I'm wandering if I can use the model I've created using this tutorial in the same way you wrote the test.py script. I have no camera or other hardware board lying around for test, but would like to know if my generated model recognises images similar to the ones I used for training (Rubber Ducks in my case :))

Any help will be appreciated!

Keep up the good work.
Arno

0
DmitryM8
DmitryM8

Reply 7 months ago

Yes, sure. For that one check the updated version of the script in my github repo, it's called mbnet_keras_args.py.
You can launch this script with --test argument and specify test folder location --test_location argument. You also need to specify dataset location(it will look for the model in the same folder where dataset folder is located. for example if your dataset is in /images/ducks, it will look for model named ducks_model.h5 in /images folder.
Good luck!

0
avdbrink
avdbrink

Reply 7 months ago

Thanks for your comment! I'll go and try that.

0
DmitryM8
DmitryM8

Tip 7 months ago

I may be slow responding to comments here, if you have an urgent issue, please post a comment on YouTube or contact me directly on LinkedIn :)
Also welcome to create issues on GitHub and pull requests, if you make useful changes to the code or upgrade the code to newer version of frameworks used(tf 2.0, nncase 0.2)!

-2019-02-23-4.35.56.png
0
experimentech
experimentech

Question 7 months ago

Current version of ncc as of writing this is 0.2
The syntax has changed.
Assuming ncc is in $path by some means on the user system, try this:

ncc compile model.tflite model.kmodel -i tflite --dataset images

0
DmitryM8
DmitryM8

Answer 7 months ago

You're right, they have changed syntax in 0.2 version, which is still beta now. I don't think Sipeed micropython firmware supports kmodel v4(generated by nncase 0.2) yet, at least not the last time I checked. So, I'm still using NNCase Converter v0.1.0 RC5.

0
KrichN
KrichN

8 months ago

Cannot convert from tflite to kmodel. Do you have any other options?