Face detection and tracking with Arduino and OpenCV

UPDATES

• Feb 20, 2013: In response to a question by student Hala Abuhasna if you wish to use the .NET Serial class, use the naming convention "\\\\.\\COMn" and replace n with a number > 9 to define your com port for COM ports above 9 such as COM10, COM11, etc.  
• Mar 23, 2012: Featured on Adafruit's Blog
• Mar 23, 2012: Featured on Interactive Design
• Dec 13, 2011: Featured on Floss For Science

This guide will also be maintained on my blog http://techbitar.blogspot.com/2012/04/face-detection-and-tracking-with.html

INTRODUCTION

In this project I have assembled a face detection and tracking system. You can see the video of the final project here:

Basically, the webcam sends video frames to OpenCV running on a Windows PC. If OpenCV detects a face it will track it and calculate its center's X,Y coordinates. The coordinates are then passed on to the Arduino via a serial USB connection. The Arduino controls the movement of the webcam with the help of two pan/tilt servos to follow the detected face.

OpenCV (Open Source Computer Vision Library: http://opencv.willowgarage.com/wiki/) is an open-source library that includes several hundreds of real-time computer vision algorithms. The OpenCV 2.x library is a C++ API.

This is an integration project between hardware and software tools. The image processing C++ code samples are provided with the openCV library and all I did was to modify the sample code for this project. I removed some of the unnecessary code and added serial communications to it so it can send X,Y values to Arduino.

CREDIT

This project would not have been possible without the team who developed OpenCV. I also benefited from Ryan Owens' tutorial (http://www.sparkfun.com/tutorials/304) which is based on Processing, OpenFrameworks, and an earlier version of OpenCV. I tried to install Processing and OpenFrameworks with no luck. So Instead, I opted for Microsoft Visual C++ 2010 Express and the latest version of OpenCV which is 2.3.1 without any middleware or wrappers.

TOOLS

Software Required

Arduino IDE 1.0 for Windows
OpenCV 2.3.1 SuperPack For Windows
Microsoft Visual C++ 2010 Express SP1
Serial C++ Library for Win32 (by Thierry Schneider)

Code Required

- OpenCV C++ (attached) techbitarFaceDetection.cpp (based on OpenCV's example facedetect.cpp)
- Arduino's (attached) cam_servo.ino (based on Ryan Owens' example SerialServoControl.pde)

Hardware Required

- PC preferably running Windows 7 SP1. The faster CPU the better.
- Arduino Uno or compatible + power source.
- Standard servos X 2.
- Webcam w/UBS interface.
- Breadboard.
- Jumper wires.
- Hobby wire to tie pan/tilt servos and webcam together.

   

Step 1: Installation and integration issues

1) Download and install the OpenCV-2.3.1-win-superpack.exe if you don't wish to deal with generating the support files yourself. Everything you need from OpenCV to build this project has already been generated in this download.

http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/

2) Download and install Microsoft Visual C++ 2010 Express

http://www.microsoft.com/visualstudio/en-us/products/2010-editions/visual-cpp-express

The OpenCV installation documentation explains how to make Visual C++ aware of the OpenCV support files (include, bin, etc). This is not a one-click job. Careful attention must be given to how Visual C++ must be configured to recognize OpenCV files.

The OpenCV team tested version 2.3.1 and Visual C++ 2010 on Windows 7 SP1.  If you are using a different configuration, be prepared for a few hiccups.

Comments

[zaqueoalejandro]

As many other people, im having troubles with implementing the library Tserial, By Thierry Schneider. Can anyone please clarify how to make the library work?, as many of us, im getting the conversion to "LPCWSTR" error.

[i.Sneh]

Where to make change for port and baud rate in Tserial.cpp Help me ...

[LGProspects]

Hey OP,

I have tried for countless hours today to get this to work but to no success. I can't get the visual/openCV part to work. I think I have narrowed it down to:

1) Not ever using visual studio or openCV before

2) Not being able to "run" visual studio

I would be extremely grateful for the instructions from the arduino code to the point of using the visual basic program.

Thanks!

[roskostyle]

Hello, I was wondering what the specs on the servo motors are? and possibly the best place to purchase them.

[khowsrow]

hey m not able to send X & Y coordinates to arduino uno using the same code with correct com ports and baud rate....can u suguest me where i m going wrong...

[gbt]

Hello,
I'm getting following error in Visual Studio 2010
Error 2 error C2664: 'CreateFileW' : cannot convert parameter 1 from 'char [10]' to 'LPCWSTR

CreateFileW function is in WinBase.h(c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Include\WinBase.h)

please help...

[sakir1990]

In your face tracking project I get errors about Tserial.
I used :

-Microsoft Windows 7 Ultimate SP1 32-Bit
-Microsoft Visual Studio 2008 Professional
-OpenCV2.3
-Tetraedre Serial Library(Tserial.h and Tserial.cpp)
-Arduino UNO R3
-Pan tilt mechanism
-Web Cam

errors ;

1>main.obj : error LNK2019: unresolved external symbol "public: void __thiscall Tserial::disconnect(void)" (?disconnect@Tserial@@QAEXXZ) referenced in function "int __cdecl _main(int,char const * *)" (?_main@@YAHHPAPBD@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: int __thiscall Tserial::connect(char *,int,enum serial_parity)" (?connect@Tserial@@QAEHPADHW4serial_parity@@@Z) referenced in function "int __cdecl _main(int,char const * *)" (?_main@@YAHHPAPBD@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: __thiscall Tserial::Tserial(void)" (??0Tserial@@QAE@XZ) referenced in function "int __cdecl _main(int,char const * *)" (?_main@@YAHHPAPBD@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: __thiscall Tserial::~Tserial(void)" (??1Tserial@@QAE@XZ) referenced in function "public: void * __thiscall Tserial::`scalar deleting destructor'(unsigned int)" (??_GTserial@@QAEPAXI@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: void __thiscall Tserial::sendChar(char)" (?sendChar@Tserial@@QAEXD@Z) referenced in function "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)


I'm going crazy, Why I get this errors? Please give me an answer..

Thanks, Good works.

[jmartizero]

Sorry in Visual C + + sends me this error when compiling in x64.
I do not understand the error.
Help me please. :)

Settings made of OpenCV: http://siddhantahuja.wordpress.com/2011/07/18/getting-started-with-opencv-2-3-in-microsoft-visual-studio-2010-in-windows-7-64-bit/

Program error:

1>------ Operación Generar iniciada: proyecto: camara_prueba_no1, configuración: Debug x64 ------
1>Compilación iniciada a las 01/12/2012 07:58:24 p. m..
1>InitializeBuildStatus:
1> Aplicando tarea Touch a "x64\Debug\camara_prueba_no1.unsuccessfulbuild".
1>ClCompile:
1> Todas las salidas están actualizadas.
1> camara_prueba_no1.cpp
1>camara_prueba_no1.cpp(95): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>camara_prueba_no1.cpp(95): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>camara_prueba_no1.cpp(96): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>camara_prueba_no1.cpp(96): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>ManifestResourceCompile:
1> Todas las salidas están actualizadas.
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: void __cdecl Tserial::disconnect(void)" (?disconnect@Tserial@@QEAAXXZ) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: bool __cdecl cv::CascadeClassifier::load(class std::basic_string,class std::allocator > const &)" (?load@CascadeClassifier@cv@@QEAA_NAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: int __cdecl Tserial::connect(char *,int,enum serial_parity)" (?connect@Tserial@@QEAAHPEADHW4serial_parity@@@Z) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: __cdecl Tserial::Tserial(void)" (??0Tserial@@QEAA@XZ) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: __cdecl Tserial::~Tserial(void)" (??1Tserial@@QEAA@XZ) sin resolver al que se hace referencia en la función "public: void * __cdecl Tserial::`scalar deleting destructor'(unsigned int)" (??_GTserial@@QEAAPEAXI@Z)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: void __cdecl Tserial::sendChar(char)" (?sendChar@Tserial@@QEAAXD@Z) sin resolver al que se hace referencia en la función "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: virtual void __cdecl cv::CascadeClassifier::detectMultiScale(class cv::Mat const &,class std::vector,class std::allocator > > &,double,int,int,class cv::Size_,class cv::Size_)" (?detectMultiScale@CascadeClassifier@cv@@UEAAXAEBVMat@2@AEAV?$vector@V?$Rect_@H@cv@@V?$allocator@V?$Rect_@H@cv@@@std@@@std@@NHHV?$Size_@H@2@2@Z) sin resolver al que se hace referencia en la función "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: __cdecl cv::CascadeClassifier::CascadeClassifier(void)" (??0CascadeClassifier@cv@@QEAA@XZ) sin resolver al que se hace referencia en la función "void __cdecl `dynamic initializer for 'face_cascade''(void)" (??__Eface_cascade@@YAXXZ)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: virtual __cdecl cv::CascadeClassifier::~CascadeClassifier(void)" (??1CascadeClassifier@cv@@UEAA@XZ) sin resolver al que se hace referencia en la función "void __cdecl `dynamic atexit destructor for 'face_cascade''(void)" (??__Fface_cascade@@YAXXZ)
1>C:\Users\marti_000\documents\visual studio 2010\Projects\camara_prueba_no1\x64\Debug\camara_prueba_no1.exe : fatal error LNK1120: 9 externos sin resolver
1>
1>ERROR al compilar.
1>
1>Tiempo transcurrido 00:00:03.71
========== Generar: 0 correctos, 1 incorrectos, 0 actualizados, 0 omitidos ==========

[rajvardhan]

Congratulations on your project. It is indeed very inspiring.
Can you please explain me that in your techbitarFaceDetection.cpp file, what is exactly being done with the variables LSB and MSB. Further, what does the following line do:
LSB = faces[i].x & 0xff;

[ElectricalEng]

Hi, i don't know what i need to make serial communication between c++ and arduino. I download Serial C++ Library for Win32 (by Thierry Schneider), I found sertest2.cpp, tserial.cpp and tserial.h, but i don't know where i have to put or include them and i don't know if there is another configrations needed. please don't skimp on the advice.

[stringstretcher]

I see the wall of eyes from Dr. Who... and want to make one :)

[techbitar]

:)

[GraffikeL]

Where is "Tserial.h"? i can't find it. It has errors

[techbitar]

I have a link to it under tools. Search for:
Serial C++ Library for Win32 (by Thierry Schneider)

[sentryGun53]

What would happen if you put a face on the camera itself.... then showed the camera a mirror?

[stringstretcher]

You think like I do... We are not alone!

[ArduArdu]

Any plans to update this for OpenCV 2.4.1?

[techbitar]

I have plans to update this guide to make it simpler to follow. In the process I will refer to the latest version of OpenCV.

[siddiqov]

Hi, i want to send x & y coordinates to arduino uno serially..i used the same code with correct com ports and baud rate but i couldn't send these values. i debugged my program and come to know that the "serial.available" value is 0. i am using visual studio express and the coding is in c++. i will appreciate for your feedback.

[Roustan]

I am having problem communicating between c++ and the arduino board. I am getting Main.obj : error LNK2019: unresolved external symbol "public: void __cdecl Tserial::disconnect(void)" (?disconnect@Tserial@@QEAAXXZ) referenced in function main.
I can see that the problem is from the serial communication but i am not sure how to solve it. Any help will be appreciated.

[mkatti]

I m not able to load cascade classifiers, it is saying error loading...
can you please tell me how to load these classifiers

[brahma shankar]

try copying haarcascade files in the project file u r making...for example if u r making project seg then copy haarcascade file in the folder Visual Studio 2010\Projects\seg\seg\...

[techbitar]

This is most likely a path issue. It's specific to your computer environment. check you visual studio preferences and ensure that it sees relevant opencv directories. it's a pain but no way around it.

[brahma shankar]

i m not able to send X & Y coordinates to arduino uno serially..i m using the same code with correct com ports and baud rate....can u suguest me where i m going wrong...

[techbitar]

Let's do this one step at a time. Forget about OpenCV for now and just write a test VB program that does nothing but send sample X,Y values and see if you can read it on the Arduino side. Let's establish that your VB setup can communicate with the Arduino. Then we will take it from there. Let me know how it goes.

[brahma shankar]

thanks for the help...nw the program is communicating..
i placed \\.\ before my com port number..

[techbitar]

Congrats!

[BrysonReece]

Is there any way to get the camera to recognize a finger or IR LED instead of a face?

[techbitar]

I have seen many such projects so it seems that it's doable. But the quickest way to have it recognize a finger is if you draw a face on it ;)

[brahma shankar]

what is the minimum requirement of the webcam ???

[techbitar]

I used a very old and discontinued USB webcam. If Windows recognizes it there's a good chance it will do the job.

[titan812]

I'm in the linux world. Is there a substitute for your Tserial.h ? thx

[Luna!]

Hi Kiteman, this is the second time I read one of your Instructable (first of with Kinect) and Congratulations! In my opinion.. One of the best around, thank you and do not stop! :)

Voted!

cya!

[techbitar]

Glad to hear that.

[techno1999]

I finally got my face tracking system up and running a couple of days ago. Your input was much appreciated.

[techbitar]

Glad to hear that.

[imakethings]

Cant you use make this using linux?

[techbitar]

OpenCV is available for Unix, Windows or Android.

[Kiteman]

[techbitar]

Thanks! I used to see a video embed button on the editor toolbar next to the link button but not any longer. I tried IE and still no difference. Don't know what I am doing wrong.