Introduction: Raspberry Pi 3 Based Simple AI Blind Navigation Cap (AIDA3)
This is a very brief and simple instructable on how to make a simple AI cap navigation device to help blind people detect if there is a human in front of them or not.I wanted to upgrade what I previously made here and here so I removed the Arduino bracelet/glove from the 1st & 2nd versions (which still can be used separately to help the blind person walk and avoid obstacles) and improved the python script written for the raspberry pi 3 to make it actually recognize -roughly- the faces and even speak to the blind person through headphones. I called the device AIDA3 since it's the 3rd version & hopefully I will improve it further in the future. If you found this instructable useful or if you found any limitations to it please let me know. If you can improve it that would be great too :)
The previous version used Arduino connected to the Raspberry Pi via USB Cable and would play a MP3 sound file when the ultrasonic sensor (attached to the Arduino) would detect an obstacle. This time I've used Raspberry Pi 3 and connected it to a USB webcam to enable it to take pictures of people and determine if there is a human in front of the blind person or not; if there is a human face it plays a sound MP3 file with recorded talking in it to the blind person thus specifying exactly what's in front of them. Note that this is face detection only NOT FACE RECOGNITION since it only detects faces. The Python code here uses a very simple Artificial Intelligence algorithm (HAAR). The theory of operation here is simple, the USB webcam detects a face using the HAAR AI algorithm, the python code instructs the pi to play the recorded MP3 file in case a human face was detected and the blind person will be able to hear it through the headphones which are attached to the raspberry pi as well. I'd like to point out quickly 2 problems I faced will making this : 1-I wanted to make the webcam record a live video feed and use that feed to detect that face since that would be more accurate than just taking a snapshot and analyzing it. However, that didn't turn out so good as it worked really really really and I mean really SLOW! I think it has something to do with pi's processing power so eventually I decided to work around the problem and just write a code that takes pictures since the multitasking isn't perfect yet (there are 2 separate codes to be written because of this but I'll explain that later), so taking the pictures saves a lot of time and saves processing power as well. Nevertheless, you're welcome to try using a webcam video feed for the face detection instead maybe it'll work better with you, if it didn't I'm sorry but that's what happened with me 2- The AI algorithm is simple and small in size and I used it because of this advantages in addition to not needing the Internet (since I live in a place with constant power, Internet and telephone failures so I had to be practical). But, this AI algorithm is also NOT 100% ACCURATE! Which means that there are some instances when it will not be able to detect some faces while they are clearly there in front of the webcam !! This is bad off course which is why if you can improve on it or tell me if you know an AI algorithm that works better that would be great (you can read more about the AI algorithm and OpenCV library which will be used in this instructable in the links in step 2). Personally I will try to improve on it in the next version. Also one of the problems of using a raspberry pi (other than the cost) is that it requires a lot of power so either use a smart phone power supply (which limits the motion obviously not what we want!) or buy a USB battery pack for raspberry pi like this one or if you have the time and/or can't afford buying a battery pack (like me) build one using a simple AA battery box (or even without it) and connect its wires to a 5v regulator and solder the regulator wires to the wires of the smart phone charger cable (the end that goes to the plug) and connect the end that goes inside the raspberry pi and it will work just as good (more on that later).
What you will need for this is :
- Raspberry Pi 3
- Micro SD Card for Raspberry Pi (16 GB or more)
- SD Card Reader (I'm assuming you have some knowledge in using Raspberry Pi and writing Python Code, but if that's not the case I've included all what you need to get started just in case)
- Keyboard & Mouse (For implementing the code phase, will be removed later)
- HDMI Cable (for connecting it to the screen while implementing the code)
- TV screen (to attach the HDMI cable to from one end and the pi from another end while implementing the code)
- 5 inch screen (TOTALLY OPTIONAL - In case you want to see the results while testing the project since it's small and portable, however you will need a separate power supply for that)
- Smart Phone Charger Power Supply for Raspberry Pi (Option 1 - not very practical)
- 5v Regulator + 4 x AA battery box + some solder wire and some duct tape & a wire cutter/stripper for connecting the wire between the Raspberry Pi jack and the regulator and the battery box which will need some soldering and covering the soldered parts with duct tape (Option 2 - In case you want to make a portable power source by yourself, a picture of one I did down there)
- 4 x AA batteries (In case you are going with option 2)
- USB battery pack for Raspberry pi (Option 3 - It will save you some time and easier to charge so it's more sustainable than Option 2 + Portable So for sure better than Option 1). I never got one but I think you can buy one from here I suppose
- USB Webcam (This Will be the all-seeing-eye)
- Headphones to be connected to the raspberry pi audio jack (I prefer the big ones since they are more comfortable to the ears)
- A Cap, a hat or any other type of head gear (Preferably something old and cheap since we're gonna make a hole in it for the camera)
Step 1: Setting Up the Pi
Believe it or not the hardest part in all of this is selecting a webcam and a set of headphones that actually work! Which is why I recommend getting cheap parts or items that you can easily return to the computer store in case it didn't work, either way it's your choice. I'll explain below how to test the headphones, sadly you won't know if the webcam is working or not unless you run the python script but usually it's a good sign if you run the command: lsusb on the terminal and it's there.
For the Raspberry Pi :
I'm assuming you have some knowledge of setting up the Raspberry Pi, formatting the SD Card, burning an ISO image of Raspbian OS on it using the SD Card reader and then powering up the Raspberry Pi using one of the power Supply options mentioned above then configuring the Raspberry Pi for the first time after connecting the display, keyboard & mouse to the pi,...etc. If you don't know how to do all that I suggest you stop reading this right now and check links like these : 1 , 2 and do some searching on each of the brief steps I mentioned above.
After successfully configuring the Raspberry Pi and turning it on, now we need to install the essential libraries that will make the scripts we write run smoothly in Python 2 (not 3). Open the terminal and enter the following commands and wait for each process to finish before entering the next command
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python-opencv
Note that for some reason the method of installing opencv library differ in some raspberry pi versions. I have no idea why is that but some may not accept that method of installtion above and requires you to enter different format like 'sudo apt-get install python-cv2' or even without the 'python' part. If such case happened search for other ways to install the opencv library using the terminal and try them all until one works with you. Connect the headphones and test to see them if they work or not by downloading any small MP3 file from the internet (ex. sound.mp3) to your pi main folder (the path of the file is important) and run the following command on the terminal
omxplayer sound.mp3
If all works fine and you hear the MP3 file clearly, connect the webcam and move on to the next step.
Step 2: The Code
Open IDLE (or any other python Compiler if you want but I just used this one) and MAKE SURE IT'S PYTHON 2 NOT 3. Here we're gonna use 2 python scripts. I tried merging them both into one script to simplify everything but I failed :(. If you can do that even better. The 2 scripts are very simple. The first script (we'll call it face_detect.py) uses the machine learning algorithm and open cv library (cv2) to detect faces and draw a rectangle on them and plays the mp3 file (using the same terminal method we used above to test the headphones) in case one or more faces were detected. The second sciprt (we'll call it image_capture.py) simply instructs the raspberry pi to take a still image from the webcam and runs the face_detect.py instantly. So simply put one script takes the image, while the other one analyzes the image to see if it has any faces or not, if there are any, it plays the mp3 file. For the speaking sound file, I used an online free text-to-speech software and I have made the sound files I made available but you can use/create any MP3 files of your choosing (you'll have to change the name of the file in the python code off course). Here are some links for the text-to-speech online free software but note that not all of them allows you to download a MP3 file
texttomp3.online
And again don't forget that the PATH OF THE FILES IS VERY IMPORTANT ! if you put the files in different folders you'll need to change the path of each one of them in the python code. I have included a screenshot of the home directory of the Raspberry Pi with the machine learning algorithm, the python scripts and the MP3 sound file.
Finally, here are some reading material If you want to learn more about HAAR Machine learning classifier and how it works with OpenCV library, these links are very informative.
http://docs.opencv.org/trunk/d7/d8b/tutorial_py_fa...
http://athenanichol.com/blog/?p=127
https://en.wikipedia.org/wiki/Cascading_classifier...
https://realpython.com/blog/python/face-recognitio...
https://realpython.com/blog/python/face-detection-...
https://opencv-python-tutroals.readthedocs.io/en/l...
http://docs.opencv.org/2.4/modules/contrib/doc/fac...
https://codeplasma.com/2012/12/03/getting-webcam-i...
Now for the python codes, I have included both of them and the mp3 file and the Machine learning algorithm as well in the attachments so you can download them and run them, but it's always better if you write the code yourself.
From the Raspberry Pi start menu choose 'programming' and choose IDLE and open a new file and write the following python code :-
import cv2
import sys
import os
faceCascade = cv2.CascadeClassifier('haarcascade_frontalface_default.xml') #machine learning model for recognizing faces
image = cv2.imread('image.jpg') #read the image
image = cv2.resize(image, (0,0), fx=0.5, fy=0.5) #resize the image to half of its current size
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) #convert the image to grayscale to ease up the process
#detect faces in the image
faces = faceCascade.detectMultiScale(gray, 1.1, 8) #(grayscale image, scale factor, minimum number of neighbours) last 1 helps in decreasing errors in face detection
print("Found {0} Human(s)!".format(len(faces)))
if len(faces) >= 1: #if number of faces detected is more than or equals one, play mp3 file
os.system('omxplayer sound.mp3')
#Draw a rectangle around the faces
for (x, y, w, h) in faces:
cv2.rectangle(image, (x, y), (x+w, y+h), (0, 255, 0), 2) # draw a rectangle around the face detected(colored image, top-left corner, bottom right corner, green color, threshold value)
cv2.imshow("Faces found", image)
cv2.waitKey(0)
cv2.destroyAllWindows()
save the previous code in a file named face_detect.py, make sure it's saved in the home directory along with the machine learning algorithm and the sound mp2 file.
Then open a new file in IDLE and write the following python code :-
import cv2
import sys
import os
ramp_frames = 30 #Number of frames to throw away while the camera adjusts to the light levels
camera = cv2.VideoCapture(0)
def get_image():
retval, im = camera.read()
return im
#Ramp the camera - these frames will be discarded and are only used to adjust light levels if necessary
for i in xrange(ramp_frames):
temp = get_image()
print("Taking Image...")
camera_capture = get_image()
file = "image.jpg"
cv2.imwrite(file, camera_capture)
del(camera)
os.system('python face_detect.py')
save the previous code under the name image_capture.py
Now for the final step which is testing the device, so far you have used a keyboard and a mouse for entering the code and arranging the files in the home directory, you can remove them and just keep the camera. Unless you wanna keep testing while moving then if you have been using a normal sized big screen maybe switch to the 5 inch screen (to continue testing and seeing the results of the code but this is completely optional)
Step 3: Putting It All Together
Use a pair of scissors to cut through the cap and make a small hole big enough for the wire of the USB webcam, insert the wire through the hole so that the webcam (the all-seeing-eye) would be in the middle of the cap (I have included some pictures of my cap after connecting everything together and I wanted to take better pictures but my camera burnt and the memory card is the only thing that I could salvage). Make sure that the webcam isn't turned upside down, connect the USB cable of the webcam to the hub in the raspberry pi and power it up, open the script and run it using F5 button, remove the keyboard and the mouse and start walking while putting the cap on your head (obviously this means that the power supply for the pi is a portable one). As I mentioned above the AI isn't very accurate and that's really frustrating since It detected a human face from a far while completely ignoring another face who was very close to the camera!! Feel free however to experiment with the face detection code using some of the examples provided in the links above and try the python codes without moving first to see which angles they work on and which ones aren't detected.
A few Important notes :
1-If you tested the sound system on your pi by playing the MP3 file separate from any other process just to make sure the Pi and the headphones are working well before running the program and you faced any problems in that area see this.
2- In case you want to make the Python program (image_capture.py) we just wrote above run as soon as the Raspberry Pi is turned on (If it weren't for the accuracy error of the AI I would recommend it but it's just an option for now) here are some useful links explaining how to do that if you don't already know 1 , 2
