loading
UPDATES
  • Feb 20, 2013: In response to a question by student Hala Abuhasna if you wish to use the .NET Serial class, use the naming convention "\\\\.\\COMn" and replace n with a number > 9 to define your com port for COM ports above 9 such as COM10, COM11, etc.  
  • Mar 23, 2012: Featured on Adafruit's Blog
  • Mar 23, 2012: Featured on Interactive Design
  • Dec 13, 2011: Featured on Floss For Science

This guide will also be maintained on my blog http://techbitar.blogspot.com/2012/04/face-detection-and-tracking-with.html

INTRODUCTION

In this project I have assembled a face detection and tracking system. You can see the video of the final project here:



Basically, the webcam sends video frames to OpenCV running on a Windows PC. If OpenCV detects a face it will track it and calculate its center's X,Y coordinates. The coordinates are then passed on to the Arduino via a serial USB connection. The Arduino controls the movement of the webcam with the help of two pan/tilt servos to follow the detected face.

OpenCV (Open Source Computer Vision Library: http://opencv.willowgarage.com/wiki/) is an open-source library that includes several hundreds of real-time computer vision algorithms. The OpenCV 2.x library is a C++ API.

This is an integration project between hardware and software tools. The image processing C++ code samples are provided with the openCV library and all I did was to modify the sample code for this project. I removed some of the unnecessary code and added serial communications to it so it can send X,Y values to Arduino.

CREDIT

This project would not have been possible without the team who developed OpenCV. I also benefited from Ryan Owens' tutorial (http://www.sparkfun.com/tutorials/304) which is based on Processing, OpenFrameworks, and an earlier version of OpenCV. I tried to install Processing and OpenFrameworks with no luck. So Instead, I opted for Microsoft Visual C++ 2010 Express and the latest version of OpenCV which is 2.3.1 without any middleware or wrappers.

TOOLS

Software Required

Arduino IDE 1.0 for Windows
OpenCV 2.3.1 SuperPack For Windows
Microsoft Visual C++ 2010 Express SP1
Serial C++ Library for Win32 (by Thierry Schneider)

Code Required

- OpenCV C++ (attached) techbitarFaceDetection.cpp (based on OpenCV's example facedetect.cpp)
- Arduino's (attached) cam_servo.ino (based on Ryan Owens' example SerialServoControl.pde)

Hardware Required

- PC preferably running Windows 7 SP1. The faster CPU the better.
- Arduino Uno or compatible + power source.
- Standard servos X 2.
- Webcam w/UBS interface.
- Breadboard.
- Jumper wires.
- Hobby wire to tie pan/tilt servos and webcam together.

   

Step 1: Installation and integration issues

1) Download and install the OpenCV-2.3.1-win-superpack.exe if you don't wish to deal with generating the support files yourself. Everything you need from OpenCV to build this project has already been generated in this download.

http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/

2) Download and install Microsoft Visual C++ 2010 Express

http://www.microsoft.com/visualstudio/en-us/products/2010-editions/visual-cpp-express

The OpenCV installation documentation explains how to make Visual C++ aware of the OpenCV support files (include, bin, etc). This is not a one-click job. Careful attention must be given to how Visual C++ must be configured to recognize OpenCV files.

The OpenCV team tested version 2.3.1 and Visual C++ 2010 on Windows 7 SP1.  If you are using a different configuration, be prepared for a few hiccups.
<p>To use this code with OpenCV3, you have to change out the way the program captures frames from the webcamera:<br><br>At the top of main, swap out &quot;CvCapture* capture;&quot; with &quot;VideoCapture capture(0);&quot;.<br>A few lines further down you find the capture attempt with: capture = cvCaptureFromCAM( 3 );, which can be removed entirely. While you're at it, remove the if sentence underneath it as well. It checks if it's able to capture with the camera, but is in reality completely pointless as if the camera isn't working you'll be getting an error before it. An error that you want, mind you, so trying to toss it aside with an if-sentence and exiting the program isn't exactly good code.<br>Next swap out the line &quot;frame = cvQueryFrame( capture );&quot; with &quot;capture &gt;&gt; frame;&quot; and you're all done.</p>
<p>Hi AndreasL8, thanks it help me with opencv 3.1. I need to change one line to make it work.</p><pre>&quot;if( capture )&quot; change with &quot;if( capture.isOpened() )&quot;</pre>
<p>I'm using a Windows 8.1 machine, running OpenCV 2.3.1 and Visual C++ Express 2010. When I compile the FE73Q6BGVXH5LPM program, I get some errors I can't figure out (I'm a C++ rookie, but am a programmer in other older languages). This is the first of two types:</p><p>1&gt;c:\users\matt\documents\visual studio 2010\projects\opencv test2\opencv test2\fe73q6bgvxh5lpm.cpp(59): error C2679: binary '=' : no operator found which takes a right-hand operand of type 'IplImage *' (or there is no acceptable conversion)</p><p>This is referring to the assignment, 'frame = cvQueryFrame( capture );' which references 'CVAPI(IplImage*) cvQueryFrame( CvCapture* capture );' out of the videoio_c.h* file. </p><p>I'm not having much luck figuring this one out. The description of this C2679 error code doesn't help much, being new to the language.</p><p>The second type (there are 4 similar to this) is. Not sure if they're fatal errors or warnings:</p><p>1&gt; c:\program files (x86)\microsoft visual studio 10.0\vc\include\opencv2\core\mat.hpp(948): could be 'cv::Mat &amp;cv::Mat::operator =(const cv::Mat &amp;)', which is referring to:</p><p>'Mat&amp; operator = (const Mat&amp; m);' in mat.hpp.</p><p>I suspect my setup and installation process may be at fault, but don't know. I did follow the instructions posted. But some further instruction or direction to go in researching this would be very much appreciated!</p><p>Matt</p>
<p>Can i use here codeblocks instead of visual studio ??</p>
<p>hi, can i do the same thing with an arduino ATmega168 ??</p>
<p>this tutorial arent useful unless you know how to install and run opencv in the pc using visual studios</p>
<p>Please help!!!. i could not correct my c2664 CreateFile() : cannot convert argument 1 from char[10] to 'LPCWSTR' problem on tserial.cpp code. Are there anyone fix this problem? Can anyone tell me exactly what should we modify in the tserial.cpp ?</p>
Yow all if you have problem with TSerial 'LPCWSTR' Error... <br>See this link.. http://stackoverflow.com/questions/3924926/cannot-convert-parameter-1-from-char-to-lpcwstr <br> <br>I got it fixed :)
<p>Hi Funkykomodo,</p><p>Although i visited the link that u wrote, i could not correct my c2664 CreateFile : cannot convert argument 1 from char[10] to 'LPCWSTR' problem on tserial.cpp code. Can you tell me exactly what you modified in the tserial.cpp ?</p>
<p>I wonder if it is doable or Raspberry Pi 2 powered by Windows 10? </p><p>It would be nice setup having Arduino + Raspberry working jointly without involvement of massive PC</p>
<p>nice topic</p>
<p>Hi! I am a grade 9 student and i would like to ask if your detector knows who is in the picture. Does this device shows the personal information of the men who is in the picture? I would like to know the answer as much as possible.. thank you so much! </p>
<p>The face tracking works well, but im stuck on making the servo rotate. I cant figure it out why it doesnt rotate. I changed the COM port on my COM3, but still no result. Any help would be greatly appreciated. Thanks</p>
<p>can u please help me to make this because i want this project as my final project but still have no idea how to make please reply must</p>
<p>has exited with code -1 (0xffffffff). <br>anyone can help me?</p>
<p>You might be using old OpenCV version. This guide is based on 2.3.1. download it here </p><p>http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/OpenCV-2.3.1-win-superpack.exe/ and dont forget to change lib names/environmental variables</p>
<p>There is one thing you'd have to correct in the part of code where COM port is identificated: if (arduino_com-&gt;connect(&quot;\\\\.\\COM13&quot;, 57600, spNONE)!=0). Usually, arduino drivers organise virtual COM's with numbers more then 9. For correct OpenFile function working you should use \\\\.\\COM13 and not just COM13.</p>
<p>I try to make this Arduino Open CV tutorial for webcams. OpenCV 3.0 <br>RC1 is compiled and installed. I also use Code::Blocks 13.12 as IDE. I <br>am a newbie what comes to OpenCV and I do not know too much about <br>Arduino either but here I am learning. Arduino IDE is 1.7.2. This <br>tutorial is made for Windows 7 and software for this OS but I bet it is <br>possible to repeat this in Ubuntu Linux 14.04 also. </p><p>When I try to compile and run the techbitarFaceDetection.cpp </p><p>the Code::Block IDE gives me following error:</p><blockquote> <br> <br>fatal error: Tserial.h: No such file or directory</blockquote><p>So please tell me what to do next and how to fix this error. </p>
<p>So, I have a question.</p><p>Arduino is just rotating the stepper motor, right? All the processing is done via C++ serial port...It's not a portable project unless I use RPi...</p>
<p>Can anybody help me out with this problem?</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::disconnect(void)&quot; (?disconnect@Tserial@@QAEXXZ) referenced in function _main</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: int __thiscall Tserial::connect(char *,int,enum serial_parity)&quot; (?connect@Tserial@@QAEHPADHW4serial_parity@@@Z) referenced in function _main</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::Tserial(void)&quot; (??0Tserial@@QAE@XZ) referenced in function _main</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::~Tserial(void)&quot; (??1Tserial@@QAE@XZ) referenced in function &quot;public: void * __thiscall Tserial::`scalar deleting destructor'(unsigned int)&quot; (??_GTserial@@QAEPAXI@Z)</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::sendChar(char)&quot; (?sendChar@Tserial@@QAEXD@Z) referenced in function &quot;void __cdecl detectAndDisplay(class cv::Mat)&quot; (?detectAndDisplay@@YAXVMat@cv@@@Z)</p>
<p>in the &quot;techbitarFaceDetection.cpp&quot; file you will include your Tserial files.</p><p>If you check, probably you will see that the only included file is Tserial.h</p><p>by adding #include &quot;Tserial.cpp&quot; you will solve this problem . This is how i solved mine</p>
<p>Thats so... CUTE :D</p>
<p>Hi, thank you so much for your helpful instructions. It really helps me. I am now at the step of <strong>OpenCV &amp; Microsoft Visual C++ Integration </strong>and I referred to the link OpenCV 2.3.1 and Visual Studio 2010</p><p><a href="http://www.deveature.com/2011/11/24/opencv-2-3-1-and-visual-studio-2010/" rel="nofollow">http://www.deveature.com/2011/11/24/opencv-2-3-1-and-visual-studio-2010/</a></p><p>I saw your comment there where you had problem with &ldquo;Cannot find or open the PDB file&rdquo;. I am now having the same problem. May I know how did you solve it?</p><p>Thank you so much for your help.</p>
<p>i fixed it using this solution - http://answers.opencv.org/question/12600/problem-in-debugging-cannot-find-or-open-the-pdb/</p>
<p>Thanks for very instructive steps. I have made it and it is so cool. </p>
Hello there...buddy i need to know some thing from u
<p>Hi,</p><p>I'm trying to do the project with Code Blocks instead of Visual Studio.</p><p>Like a lot of people, I have an error with tserial.h: &quot;undefined reference to Tserial::Tserial().</p><p>I've added the library to the path. Do you have any idea of the problem?</p><p>Thank you very much!! </p>
<p>Hi, in the arduino file`s &quot;#include&quot;(i`m new on this), in your code, i don`t see what`s included anywhere. Can someone help on this?</p><p>Thanks</p>
Hi, i don't know what i need to make serial communication between c++ and arduino. I download Serial C++ Library for Win32 (by Thierry Schneider), I found sertest2.cpp, tserial.cpp and tserial.h, but i don't know where i have to put or include them and i don't know if there is another configrations needed. please don't skimp on the advice.
put all of them in your project folder..
Thanks for the tutorial..After countless hours testing, i made it..pheww.. <br>anyway, to avoid error, don't forget to put Tserial.cpp and Tserial.h in your project folder..that should fix LNK2019 problem i guess..
hmm, bounding box issue? <br>
I have everything wired up, fixed the Tserial issue, fixed the LINK2019 and TBB_dll issues. Now I am gettign this error message. <br> <br>The program '[10936] techbitarFaceDetection.exe: Native' has exited with code -1073741701 (0xc000007b). <br> <br>I am stumped. Does anybody have any idea why this is happening? <br>
it would cause a little bit trouble if you are doing with 64-bit system. because C++ 2010 express could only compile with *64 compiler which is contained in Windows 7.1SDK. SDK always fail me.
Btw it for me the servo were run too slow.. not as fast as it is in the video. <br> <br>Anybody here know how to make the servos run smoothly?
As many other people, im having troubles with implementing the library Tserial, By Thierry Schneider. Can anyone please clarify how to make the library work?, as many of us, im getting the conversion to &quot;LPCWSTR&quot; error.
Where to make change for port and baud rate in Tserial.cpp Help me ...
Hey OP, <br> <br>I have tried for countless hours today to get this to work but to no success. I can't get the visual/openCV part to work. I think I have narrowed it down to: <br> <br>1) Not ever using visual studio or openCV before <br> <br>2) Not being able to &quot;run&quot; visual studio <br> <br>I would be extremely grateful for the instructions from the arduino code to the point of using the visual basic program. <br> <br>Thanks!
Hello, I was wondering what the specs on the servo motors are? and possibly the best place to purchase them.
hey m not able to send X &amp; Y coordinates to arduino uno using the same code with correct com ports and baud rate....can u suguest me where i m going wrong...
Hello, <br>I'm getting following error in Visual Studio 2010 <br>Error 2 error C2664: 'CreateFileW' : cannot convert parameter 1 from 'char [10]' to 'LPCWSTR <br> <br>CreateFileW function is in WinBase.h(c:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\Include\WinBase.h) <br> <br>please help... <br> <br>
In your face tracking project I get errors about Tserial. <br>I used : <br> <br>-Microsoft Windows 7 Ultimate SP1 32-Bit <br>-Microsoft Visual Studio 2008 Professional <br>-OpenCV2.3 <br>-Tetraedre Serial Library(Tserial.h and Tserial.cpp) <br>-Arduino UNO R3 <br>-Pan tilt mechanism <br>-Web Cam <br> <br>errors ; <br> <br>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::disconnect(void)&quot; (?disconnect@Tserial@@QAEXXZ) referenced in function &quot;int __cdecl _main(int,char const * *)&quot; (?_main@@YAHHPAPBD@Z) <br>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: int __thiscall Tserial::connect(char *,int,enum serial_parity)&quot; (?connect@Tserial@@QAEHPADHW4serial_parity@@@Z) referenced in function &quot;int __cdecl _main(int,char const * *)&quot; (?_main@@YAHHPAPBD@Z) <br>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::Tserial(void)&quot; (??0Tserial@@QAE@XZ) referenced in function &quot;int __cdecl _main(int,char const * *)&quot; (?_main@@YAHHPAPBD@Z) <br>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::~Tserial(void)&quot; (??1Tserial@@QAE@XZ) referenced in function &quot;public: void * __thiscall Tserial::`scalar deleting destructor'(unsigned int)&quot; (??_GTserial@@QAEPAXI@Z) <br>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::sendChar(char)&quot; (?sendChar@Tserial@@QAEXD@Z) referenced in function &quot;void __cdecl detectAndDisplay(class cv::Mat)&quot; (?detectAndDisplay@@YAXVMat@cv@@@Z) <br> <br> <br>I'm going crazy, Why I get this errors? Please give me an answer.. <br> <br>Thanks, Good works.
Sorry in Visual C + + sends me this error when compiling in x64. <br>I do not understand the error. <br>Help me please. :) <br> <br>Settings made of OpenCV: http://siddhantahuja.wordpress.com/2011/07/18/getting-started-with-opencv-2-3-in-microsoft-visual-studio-2010-in-windows-7-64-bit/ <br> <br>Program error: <br> <br>1&gt;------ Operaci&oacute;n Generar iniciada: proyecto: camara_prueba_no1, configuraci&oacute;n: Debug x64 ------ <br>1&gt;Compilaci&oacute;n iniciada a las 01/12/2012 07:58:24 p. m.. <br>1&gt;InitializeBuildStatus: <br>1&gt; Aplicando tarea Touch a &quot;x64\Debug\camara_prueba_no1.unsuccessfulbuild&quot;. <br>1&gt;ClCompile: <br>1&gt; Todas las salidas est&aacute;n actualizadas. <br>1&gt; camara_prueba_no1.cpp <br>1&gt;camara_prueba_no1.cpp(95): warning C4244: 'argumento' : conversi&oacute;n de 'double' a 'int'; posible p&eacute;rdida de datos <br>1&gt;camara_prueba_no1.cpp(95): warning C4244: 'argumento' : conversi&oacute;n de 'double' a 'int'; posible p&eacute;rdida de datos <br>1&gt;camara_prueba_no1.cpp(96): warning C4244: 'argumento' : conversi&oacute;n de 'double' a 'int'; posible p&eacute;rdida de datos <br>1&gt;camara_prueba_no1.cpp(96): warning C4244: 'argumento' : conversi&oacute;n de 'double' a 'int'; posible p&eacute;rdida de datos <br>1&gt;ManifestResourceCompile: <br>1&gt; Todas las salidas est&aacute;n actualizadas. <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: void __cdecl Tserial::disconnect(void)&quot; (?disconnect@Tserial@@QEAAXXZ) sin resolver al que se hace referencia en la funci&oacute;n main <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: bool __cdecl cv::CascadeClassifier::load(class std::basic_string,class std::allocator &gt; const &amp;)&quot; (?load@CascadeClassifier@cv@@QEAA_NAEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) sin resolver al que se hace referencia en la funci&oacute;n main <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: int __cdecl Tserial::connect(char *,int,enum serial_parity)&quot; (?connect@Tserial@@QEAAHPEADHW4serial_parity@@@Z) sin resolver al que se hace referencia en la funci&oacute;n main <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: __cdecl Tserial::Tserial(void)&quot; (??0Tserial@@QEAA@XZ) sin resolver al que se hace referencia en la funci&oacute;n main <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: __cdecl Tserial::~Tserial(void)&quot; (??1Tserial@@QEAA@XZ) sin resolver al que se hace referencia en la funci&oacute;n &quot;public: void * __cdecl Tserial::`scalar deleting destructor'(unsigned int)&quot; (??_GTserial@@QEAAPEAXI@Z) <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: void __cdecl Tserial::sendChar(char)&quot; (?sendChar@Tserial@@QEAAXD@Z) sin resolver al que se hace referencia en la funci&oacute;n &quot;void __cdecl detectAndDisplay(class cv::Mat)&quot; (?detectAndDisplay@@YAXVMat@cv@@@Z) <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: virtual void __cdecl cv::CascadeClassifier::detectMultiScale(class cv::Mat const &amp;,class std::vector,class std::allocator &gt; &gt; &amp;,double,int,int,class cv::Size_,class cv::Size_)&quot; (?detectMultiScale@CascadeClassifier@cv@@UEAAXAEBVMat@2@AEAV?$vector@V?$Rect_@H@cv@@V?$allocator@V?$Rect_@H@cv@@@std@@@std@@NHHV?$Size_@H@2@2@Z) sin resolver al que se hace referencia en la funci&oacute;n &quot;void __cdecl detectAndDisplay(class cv::Mat)&quot; (?detectAndDisplay@@YAXVMat@cv@@@Z) <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: __cdecl cv::CascadeClassifier::CascadeClassifier(void)&quot; (??0CascadeClassifier@cv@@QEAA@XZ) sin resolver al que se hace referencia en la funci&oacute;n &quot;void __cdecl `dynamic initializer for 'face_cascade''(void)&quot; (??__Eface_cascade@@YAXXZ) <br>1&gt;camara_prueba_no1.obj : error LNK2019: s&iacute;mbolo externo &quot;public: virtual __cdecl cv::CascadeClassifier::~CascadeClassifier(void)&quot; (??1CascadeClassifier@cv@@UEAA@XZ) sin resolver al que se hace referencia en la funci&oacute;n &quot;void __cdecl `dynamic atexit destructor for 'face_cascade''(void)&quot; (??__Fface_cascade@@YAXXZ) <br>1&gt;C:\Users\marti_000\documents\visual studio 2010\Projects\camara_prueba_no1\x64\Debug\camara_prueba_no1.exe : fatal error LNK1120: 9 externos sin resolver <br>1&gt; <br>1&gt;ERROR al compilar. <br>1&gt; <br>1&gt;Tiempo transcurrido 00:00:03.71 <br>========== Generar: 0 correctos, 1 incorrectos, 0 actualizados, 0 omitidos ========== <br>
Congratulations on your project. It is indeed very inspiring. <br>Can you please explain me that in your techbitarFaceDetection.cpp file, what is exactly being done with the variables LSB and MSB. Further, what does the following line do: <br>LSB = faces[i].x &amp; 0xff; <br>
I see the wall of eyes from Dr. Who... and want to make one :)
:)
Where is &quot;Tserial.h&quot;? i can't find it. It has errors
I have a link to it under tools. Search for: <br>Serial C++ Library for Win32 (by Thierry Schneider)
What would happen if you put a face on the camera itself.... then showed the camera a mirror?
You think like I do... We are not alone!

About This Instructable

280,910 views

503 favorites

License:

Bio: Did I unplug the solder iron?
More by techbitar: IR Remote Control Station for Android - TURN THE TV DOWN SensoDuino: Turn Your Android Phone into a Wireless Sensors Hub for Arduino Modify The HC-05 Bluetooth Module Defaults Using AT Commands
Add instructable to: