Instructables

Face detection and tracking with Arduino and OpenCV

UPDATES
  • Feb 20, 2013: In response to a question by student Hala Abuhasna if you wish to use the .NET Serial class, use the naming convention "\\\\.\\COMn" and replace n with a number > 9 to define your com port for COM ports above 9 such as COM10, COM11, etc.  
  • Mar 23, 2012: Featured on Adafruit's Blog
  • Mar 23, 2012: Featured on Interactive Design
  • Dec 13, 2011: Featured on Floss For Science

This guide will also be maintained on my blog http://techbitar.blogspot.com/2012/04/face-detection-and-tracking-with.html

INTRODUCTION

In this project I have assembled a face detection and tracking system. You can see the video of the final project here:



Basically, the webcam sends video frames to OpenCV running on a Windows PC. If OpenCV detects a face it will track it and calculate its center's X,Y coordinates. The coordinates are then passed on to the Arduino via a serial USB connection. The Arduino controls the movement of the webcam with the help of two pan/tilt servos to follow the detected face.

OpenCV (Open Source Computer Vision Library: http://opencv.willowgarage.com/wiki/) is an open-source library that includes several hundreds of real-time computer vision algorithms. The OpenCV 2.x library is a C++ API.

This is an integration project between hardware and software tools. The image processing C++ code samples are provided with the openCV library and all I did was to modify the sample code for this project. I removed some of the unnecessary code and added serial communications to it so it can send X,Y values to Arduino.

CREDIT

This project would not have been possible without the team who developed OpenCV. I also benefited from Ryan Owens' tutorial (http://www.sparkfun.com/tutorials/304) which is based on Processing, OpenFrameworks, and an earlier version of OpenCV. I tried to install Processing and OpenFrameworks with no luck. So Instead, I opted for Microsoft Visual C++ 2010 Express and the latest version of OpenCV which is 2.3.1 without any middleware or wrappers.

TOOLS

Software Required

Arduino IDE 1.0 for Windows
OpenCV 2.3.1 SuperPack For Windows
Microsoft Visual C++ 2010 Express SP1
Serial C++ Library for Win32 (by Thierry Schneider)

Code Required

- OpenCV C++ (attached) techbitarFaceDetection.cpp (based on OpenCV's example facedetect.cpp)
- Arduino's (attached) cam_servo.ino (based on Ryan Owens' example SerialServoControl.pde)

Hardware Required

- PC preferably running Windows 7 SP1. The faster CPU the better.
- Arduino Uno or compatible + power source.
- Standard servos X 2.
- Webcam w/UBS interface.
- Breadboard.
- Jumper wires.
- Hobby wire to tie pan/tilt servos and webcam together.

   
 
Remove these adsRemove these ads by Signing Up
1-40 of 52Next »
dada828 days ago

Hi,

I'm trying to do the project with Code Blocks instead of Visual Studio.

Like a lot of people, I have an error with tserial.h: "undefined reference to Tserial::Tserial().

I've added the library to the path. Do you have any idea of the problem?

Thank you very much!!

sinao5 months ago

Hi, in the arduino file`s "#include"(i`m new on this), in your code, i don`t see what`s included anywhere. Can someone help on this?

Thanks

dnarayanan935 months ago

Can anybody help me out with this problem?

1>facedetect.obj : error LNK2019: unresolved external symbol "public: void __thiscall Tserial::disconnect(void)" (?disconnect@Tserial@@QAEXXZ) referenced in function _main

1>facedetect.obj : error LNK2019: unresolved external symbol "public: int __thiscall Tserial::connect(char *,int,enum serial_parity)" (?connect@Tserial@@QAEHPADHW4serial_parity@@@Z) referenced in function _main

1>facedetect.obj : error LNK2019: unresolved external symbol "public: __thiscall Tserial::Tserial(void)" (??0Tserial@@QAE@XZ) referenced in function _main

1>facedetect.obj : error LNK2019: unresolved external symbol "public: __thiscall Tserial::~Tserial(void)" (??1Tserial@@QAE@XZ) referenced in function "public: void * __thiscall Tserial::`scalar deleting destructor'(unsigned int)" (??_GTserial@@QAEPAXI@Z)

1>facedetect.obj : error LNK2019: unresolved external symbol "public: void __thiscall Tserial::sendChar(char)" (?sendChar@Tserial@@QAEXD@Z) referenced in function "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

swapyuee8 months ago

Thanks for very instructive steps. I have made it and it is so cool.

Hi, i don't know what i need to make serial communication between c++ and arduino. I download Serial C++ Library for Win32 (by Thierry Schneider), I found sertest2.cpp, tserial.cpp and tserial.h, but i don't know where i have to put or include them and i don't know if there is another configrations needed. please don't skimp on the advice.
put all of them in your project folder..
dbin din1 year ago
Thanks for the tutorial..After countless hours testing, i made it..pheww..
anyway, to avoid error, don't forget to put Tserial.cpp and Tserial.h in your project folder..that should fix LNK2019 problem i guess..
ok.PNGok.PNG
hmm, bounding box issue?
roskostyle1 year ago
I have everything wired up, fixed the Tserial issue, fixed the LINK2019 and TBB_dll issues. Now I am gettign this error message.

The program '[10936] techbitarFaceDetection.exe: Native' has exited with code -1073741701 (0xc000007b).

I am stumped. Does anybody have any idea why this is happening?
vivaroma1 year ago
it would cause a little bit trouble if you are doing with 64-bit system. because C++ 2010 express could only compile with *64 compiler which is contained in Windows 7.1SDK. SDK always fail me.
Btw it for me the servo were run too slow.. not as fast as it is in the video.

Anybody here know how to make the servos run smoothly?
Yow all if you have problem with TSerial 'LPCWSTR' Error...
See this link.. http://stackoverflow.com/questions/3924926/cannot-convert-parameter-1-from-char-to-lpcwstr

I got it fixed :)
As many other people, im having troubles with implementing the library Tserial, By Thierry Schneider. Can anyone please clarify how to make the library work?, as many of us, im getting the conversion to "LPCWSTR" error.
i.Sneh1 year ago
Where to make change for port and baud rate in Tserial.cpp Help me ...
Hey OP,

I have tried for countless hours today to get this to work but to no success. I can't get the visual/openCV part to work. I think I have narrowed it down to:

1) Not ever using visual studio or openCV before

2) Not being able to "run" visual studio

I would be extremely grateful for the instructions from the arduino code to the point of using the visual basic program.

Thanks!
roskostyle1 year ago
Hello, I was wondering what the specs on the servo motors are? and possibly the best place to purchase them.
khowsrow1 year ago
hey m not able to send X & Y coordinates to arduino uno using the same code with correct com ports and baud rate....can u suguest me where i m going wrong...
gbt1 year ago
Hello,
I'm getting following error in Visual Studio 2010
Error 2 error C2664: 'CreateFileW' : cannot convert parameter 1 from 'char [10]' to 'LPCWSTR

CreateFileW function is in WinBase.h(c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Include\WinBase.h)

please help...

sakir19901 year ago
In your face tracking project I get errors about Tserial.
I used :

-Microsoft Windows 7 Ultimate SP1 32-Bit
-Microsoft Visual Studio 2008 Professional
-OpenCV2.3
-Tetraedre Serial Library(Tserial.h and Tserial.cpp)
-Arduino UNO R3
-Pan tilt mechanism
-Web Cam

errors ;

1>main.obj : error LNK2019: unresolved external symbol "public: void __thiscall Tserial::disconnect(void)" (?disconnect@Tserial@@QAEXXZ) referenced in function "int __cdecl _main(int,char const * *)" (?_main@@YAHHPAPBD@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: int __thiscall Tserial::connect(char *,int,enum serial_parity)" (?connect@Tserial@@QAEHPADHW4serial_parity@@@Z) referenced in function "int __cdecl _main(int,char const * *)" (?_main@@YAHHPAPBD@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: __thiscall Tserial::Tserial(void)" (??0Tserial@@QAE@XZ) referenced in function "int __cdecl _main(int,char const * *)" (?_main@@YAHHPAPBD@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: __thiscall Tserial::~Tserial(void)" (??1Tserial@@QAE@XZ) referenced in function "public: void * __thiscall Tserial::`scalar deleting destructor'(unsigned int)" (??_GTserial@@QAEPAXI@Z)
1>main.obj : error LNK2019: unresolved external symbol "public: void __thiscall Tserial::sendChar(char)" (?sendChar@Tserial@@QAEXD@Z) referenced in function "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)


I'm going crazy, Why I get this errors? Please give me an answer..

Thanks, Good works.
jmartizero1 year ago
Sorry in Visual C + + sends me this error when compiling in x64.
I do not understand the error.
Help me please. :)

Settings made of OpenCV: http://siddhantahuja.wordpress.com/2011/07/18/getting-started-with-opencv-2-3-in-microsoft-visual-studio-2010-in-windows-7-64-bit/

Program error:

1>------ Operación Generar iniciada: proyecto: camara_prueba_no1, configuración: Debug x64 ------
1>Compilación iniciada a las 01/12/2012 07:58:24 p. m..
1>InitializeBuildStatus:
1> Aplicando tarea Touch a "x64\Debug\camara_prueba_no1.unsuccessfulbuild".
1>ClCompile:
1> Todas las salidas están actualizadas.
1> camara_prueba_no1.cpp
1>camara_prueba_no1.cpp(95): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>camara_prueba_no1.cpp(95): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>camara_prueba_no1.cpp(96): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>camara_prueba_no1.cpp(96): warning C4244: 'argumento' : conversión de 'double' a 'int'; posible pérdida de datos
1>ManifestResourceCompile:
1> Todas las salidas están actualizadas.
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: void __cdecl Tserial::disconnect(void)" (?disconnect@Tserial@@QEAAXXZ) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: bool __cdecl cv::CascadeClassifier::load(class std::basic_string,class std::allocator > const &)" (?load@CascadeClassifier@cv@@QEAA_NAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: int __cdecl Tserial::connect(char *,int,enum serial_parity)" (?connect@Tserial@@QEAAHPEADHW4serial_parity@@@Z) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: __cdecl Tserial::Tserial(void)" (??0Tserial@@QEAA@XZ) sin resolver al que se hace referencia en la función main
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: __cdecl Tserial::~Tserial(void)" (??1Tserial@@QEAA@XZ) sin resolver al que se hace referencia en la función "public: void * __cdecl Tserial::`scalar deleting destructor'(unsigned int)" (??_GTserial@@QEAAPEAXI@Z)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: void __cdecl Tserial::sendChar(char)" (?sendChar@Tserial@@QEAAXD@Z) sin resolver al que se hace referencia en la función "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: virtual void __cdecl cv::CascadeClassifier::detectMultiScale(class cv::Mat const &,class std::vector,class std::allocator > > &,double,int,int,class cv::Size_,class cv::Size_)" (?detectMultiScale@CascadeClassifier@cv@@UEAAXAEBVMat@2@AEAV?$vector@V?$Rect_@H@cv@@V?$allocator@V?$Rect_@H@cv@@@std@@@std@@NHHV?$Size_@H@2@2@Z) sin resolver al que se hace referencia en la función "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: __cdecl cv::CascadeClassifier::CascadeClassifier(void)" (??0CascadeClassifier@cv@@QEAA@XZ) sin resolver al que se hace referencia en la función "void __cdecl `dynamic initializer for 'face_cascade''(void)" (??__Eface_cascade@@YAXXZ)
1>camara_prueba_no1.obj : error LNK2019: símbolo externo "public: virtual __cdecl cv::CascadeClassifier::~CascadeClassifier(void)" (??1CascadeClassifier@cv@@UEAA@XZ) sin resolver al que se hace referencia en la función "void __cdecl `dynamic atexit destructor for 'face_cascade''(void)" (??__Fface_cascade@@YAXXZ)
1>C:\Users\marti_000\documents\visual studio 2010\Projects\camara_prueba_no1\x64\Debug\camara_prueba_no1.exe : fatal error LNK1120: 9 externos sin resolver
1>
1>ERROR al compilar.
1>
1>Tiempo transcurrido 00:00:03.71
========== Generar: 0 correctos, 1 incorrectos, 0 actualizados, 0 omitidos ==========
rajvardhan1 year ago
Congratulations on your project. It is indeed very inspiring.
Can you please explain me that in your techbitarFaceDetection.cpp file, what is exactly being done with the variables LSB and MSB. Further, what does the following line do:
LSB = faces[i].x & 0xff;
I see the wall of eyes from Dr. Who... and want to make one :)
techbitar (author)  stringstretcher2 years ago
:)
GraffikeL2 years ago
Where is "Tserial.h"? i can't find it. It has errors
techbitar (author)  GraffikeL2 years ago
I have a link to it under tools. Search for:
Serial C++ Library for Win32 (by Thierry Schneider)
sentryGun532 years ago
What would happen if you put a face on the camera itself.... then showed the camera a mirror?
You think like I do... We are not alone!
ArduArdu2 years ago
Any plans to update this for OpenCV 2.4.1?
techbitar (author)  ArduArdu2 years ago
I have plans to update this guide to make it simpler to follow. In the process I will refer to the latest version of OpenCV.
siddiqov2 years ago
Hi, i want to send x & y coordinates to arduino uno serially..i used the same code with correct com ports and baud rate but i couldn't send these values. i debugged my program and come to know that the "serial.available" value is 0. i am using visual studio express and the coding is in c++. i will appreciate for your feedback.
Roustan2 years ago
I am having problem communicating between c++ and the arduino board. I am getting Main.obj : error LNK2019: unresolved external symbol "public: void __cdecl Tserial::disconnect(void)" (?disconnect@Tserial@@QEAAXXZ) referenced in function main.
I can see that the problem is from the serial communication but i am not sure how to solve it. Any help will be appreciated.
mkatti2 years ago
I m not able to load cascade classifiers, it is saying error loading...
can you please tell me how to load these classifiers
try copying haarcascade files in the project file u r making...for example if u r making project seg then copy haarcascade file in the folder Visual Studio 2010\Projects\seg\seg\...
techbitar (author)  mkatti2 years ago
This is most likely a path issue. It's specific to your computer environment. check you visual studio preferences and ensure that it sees relevant opencv directories. it's a pain but no way around it.
i m not able to send X & Y coordinates to arduino uno serially..i m using the same code with correct com ports and baud rate....can u suguest me where i m going wrong...
techbitar (author)  brahma shankar2 years ago
Let's do this one step at a time. Forget about OpenCV for now and just write a test VB program that does nothing but send sample X,Y values and see if you can read it on the Arduino side. Let's establish that your VB setup can communicate with the Arduino. Then we will take it from there. Let me know how it goes.
thanks for the help...nw the program is communicating..
i placed \\.\ before my com port number..
techbitar (author)  brahma shankar2 years ago
Congrats!
BrysonReece2 years ago
Is there any way to get the camera to recognize a finger or IR LED instead of a face?
techbitar (author)  BrysonReece2 years ago
I have seen many such projects so it seems that it's doable. But the quickest way to have it recognize a finger is if you draw a face on it ;)
1-40 of 52Next »