• Feb 20, 2013: In response to a question by student Hala Abuhasna if you wish to use the .NET Serial class, use the naming convention "\\\\.\\COMn" and replace n with a number > 9 to define your com port for COM ports above 9 such as COM10, COM11, etc.  
  • Mar 23, 2012: Featured on Adafruit's Blog
  • Mar 23, 2012: Featured on Interactive Design
  • Dec 13, 2011: Featured on Floss For Science

This guide will also be maintained on my blog http://techbitar.blogspot.com/2012/04/face-detection-and-tracking-with.html


In this project I have assembled a face detection and tracking system. You can see the video of the final project here:

Basically, the webcam sends video frames to OpenCV running on a Windows PC. If OpenCV detects a face it will track it and calculate its center's X,Y coordinates. The coordinates are then passed on to the Arduino via a serial USB connection. The Arduino controls the movement of the webcam with the help of two pan/tilt servos to follow the detected face.

OpenCV (Open Source Computer Vision Library: http://opencv.willowgarage.com/wiki/) is an open-source library that includes several hundreds of real-time computer vision algorithms. The OpenCV 2.x library is a C++ API.

This is an integration project between hardware and software tools. The image processing C++ code samples are provided with the openCV library and all I did was to modify the sample code for this project. I removed some of the unnecessary code and added serial communications to it so it can send X,Y values to Arduino.


This project would not have been possible without the team who developed OpenCV. I also benefited from Ryan Owens' tutorial (http://www.sparkfun.com/tutorials/304) which is based on Processing, OpenFrameworks, and an earlier version of OpenCV. I tried to install Processing and OpenFrameworks with no luck. So Instead, I opted for Microsoft Visual C++ 2010 Express and the latest version of OpenCV which is 2.3.1 without any middleware or wrappers.


Software Required

Arduino IDE 1.0 for Windows
OpenCV 2.3.1 SuperPack For Windows
Microsoft Visual C++ 2010 Express SP1
Serial C++ Library for Win32 (by Thierry Schneider)

Code Required

- OpenCV C++ (attached) techbitarFaceDetection.cpp (based on OpenCV's example facedetect.cpp)
- Arduino's (attached) cam_servo.ino (based on Ryan Owens' example SerialServoControl.pde)

Hardware Required

- PC preferably running Windows 7 SP1. The faster CPU the better.
- Arduino Uno or compatible + power source.
- Standard servos X 2.
- Webcam w/UBS interface.
- Breadboard.
- Jumper wires.
- Hobby wire to tie pan/tilt servos and webcam together.


Step 1: Installation and Integration Issues

1) Download and install the OpenCV-2.3.1-win-superpack.exe if you don't wish to deal with generating the support files yourself. Everything you need from OpenCV to build this project has already been generated in this download.


2) Download and install Microsoft Visual C++ 2010 Express


The OpenCV installation documentation explains how to make Visual C++ aware of the OpenCV support files (include, bin, etc). This is not a one-click job. Careful attention must be given to how Visual C++ must be configured to recognize OpenCV files.

The OpenCV team tested version 2.3.1 and Visual C++ 2010 on Windows 7 SP1.  If you are using a different configuration, be prepared for a few hiccups.

Step 2: Attach Servos and Camera

I did not want to affix any of the project parts permanently because I like to take my projects apart after I am done.  So I used hobby wire, which is nothing more than a stiff wire, to tie the servos and the webcam together.

I wrapped the base of the webcam to the pan servo horn. Then I wrapped a cable around the horn of the tilt servo and the body of the pan servo.

To keep the whole servo/webcam assembly from moving randomly during operation, I used a clamp to tie it down to the soldering helping hands. It kept is steady during servo rotation.

It ain't pretty but it works.

Step 3: Wiring the Parts Together

The wiring is straight forward. I used a breadboard to make the connections.

The yellow/signal wire for the pan (X axis) servo goes to digital pin 9.
The yellow/signal wire for the tilt (y axis) servo goes to digital pin 10.
The red/Vcc wires of both servos go to the Arduino's 5V pin.
The black/GND wires of both servos go to Arduino's GND pin.

The webcam's USB goes to the PC. The C++ code will identify it via a number representing the USB port its connected to. .

The Arduino Uno is connected to the PC via USB. Take note of the COM port the USB is connected to. You can find the com port from the Arduino Tools/Serial Ports menu. You will see a check mark next to the active USB port. This is the com port that you will use in your C++ code to communicate with Arduino.

You must change the C++ code to match the PC's com port and baud rate with Arduino's. Also, the C++ code must be told which USB port the webcam is using.

Step 4: Resources

I found these websites to be informative:

OpenCV Face Detection/Tracking Guides

Face Tracking with a Pan/Tilt Servo Bracket by zagGrad

Arduino + Servo + openCV Tutorial [openFrameworks]

OpenCV & Microsoft Visual C++ Integration

OpenCV 2.3.1 and Visual Studio 2010

Getting Started with OpenCV 2.3 in Microsoft Visual Studio 2010 in Windows

OpenCV 2.1.0 with Visual Studio 2010

How to build applications with OpenCV inside the Microsoft Visual Studio

Using OpenCV 2.3.1 with Visual Studio 2010 (tutorial)
<p>this tutorial is not working with visual studio 2017 so would you make a tutorial with lastest version of visual studio please ( not with visual studio 2010, it is too old ) ? tyvm</p>
<p>Hello everyone!</p><p>Someone was able to correct the error: can not convert argument 1 from char [10] to 'LPCWSTR'?</p><p>Help me please!</p>
<p>hi gooooooooooood jobbb....but i like to do this with a cart raspberry pi 2 and </p><p>step-by-step engine......plzzzz samoane help me :( :( :(</p>
<p>hi I wan to write a code in general to detect any object of any size is it possible?</p>
Yow all if you have problem with TSerial 'LPCWSTR' Error... <br>See this link.. http://stackoverflow.com/questions/3924926/cannot-convert-parameter-1-from-char-to-lpcwstr <br> <br>I got it fixed :)
<p>Hi Funkykomodo,</p><p>Although i visited the link that u wrote, i could not correct my c2664 CreateFile : cannot convert argument 1 from char[10] to 'LPCWSTR' problem on tserial.cpp code. Can you tell me exactly what you modified in the tserial.cpp ?</p>
<p>I commented out </p><p>//#include &lt;stdafx.h&gt;</p><p>and then i changed</p><p>serial_handle = CreateFile(port, GENERIC_READ | GENERIC_WRITE,</p><p> 0, NULL, OPEN_EXISTING,NULL,NULL);</p><p>to</p><p>serial_handle = CreateFileA(port, GENERIC_READ | GENERIC_WRITE,</p><p> 0, NULL, OPEN_EXISTING,NULL,NULL);</p><p>Unfortunately, nothing is happening on my serial monitor yet</p>
<p>Hi </p><p>I want to make a cam to recognize things by shape and color of it without using PC or computer (just arduino).</p><p>please give me a URL or something to help.</p><p>thanks</p>
<p>hi...this my first project in opencv and arduino and i am struck with this error can you solve this.i followed everthing step by step and i'm finding some difficulties in this please solve this</p><p>1&gt;------ Build started: Project: f_tr, Configuration: Debug Win32 ------</p><p>1&gt;Build started 19/11/2016 20:51:20.</p><p>1&gt;InitializeBuildStatus:</p><p>1&gt; Touching &quot;Debug\f_tr.unsuccessfulbuild&quot;.</p><p>1&gt;ClCompile:</p><p>1&gt; All outputs are up-to-date.</p><p>1&gt;ManifestResourceCompile:</p><p>1&gt; All outputs are up-to-date.</p><p>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: bool __thiscall cv::CascadeClassifier::load(class std::basic_string&lt;char,struct std::char_traits&lt;char&gt;,class std::allocator&lt;char&gt; &gt; const &amp;)&quot; (?load@CascadeClassifier@cv@@QAE_NABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) referenced in function _main</p><p>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: virtual void __thiscall cv::CascadeClassifier::detectMultiScale(class cv::Mat const &amp;,class std::vector&lt;class cv::Rect_&lt;int&gt;,class std::allocator&lt;class cv::Rect_&lt;int&gt; &gt; &gt; &amp;,double,int,int,class cv::Size_&lt;int&gt;,class cv::Size_&lt;int&gt;)&quot; (?detectMultiScale@CascadeClassifier@cv@@UAEXABVMat@2@AAV?$vector@V?$Rect_@H@cv@@V?$allocator@V?$Rect_@H@cv@@@std@@@std@@NHHV?$Size_@H@2@2@Z) referenced in function &quot;void __cdecl detectAndDisplay(class cv::Mat)&quot; (?detectAndDisplay@@YAXVMat@cv@@@Z)</p><p>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall cv::CascadeClassifier::CascadeClassifier(void)&quot; (??0CascadeClassifier@cv@@QAE@XZ) referenced in function &quot;void __cdecl `dynamic initializer for 'face_cascade''(void)&quot; (??__Eface_cascade@@YAXXZ)</p><p>1&gt;main.obj : error LNK2019: unresolved external symbol &quot;public: virtual __thiscall cv::CascadeClassifier::~CascadeClassifier(void)&quot; (??1CascadeClassifier@cv@@UAE@XZ) referenced in function &quot;void __cdecl `dynamic atexit destructor for 'face_cascade''(void)&quot; (??__Fface_cascade@@YAXXZ)</p><p>1&gt;C:\Users\Marshal Rev\documents\visual studio 2010\Projects\f_tr\Debug\f_tr.exe : fatal error LNK1120: 4 unresolved externals</p><p>1&gt;</p><p>1&gt;Build FAILED.</p><p>1&gt;</p><p>1&gt;Time Elapsed 00:00:00.19</p><p>========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========</p>
<p>hi! I don &acute;t have any errors when building it, the compilation have succeeded, but it doesn&acute;t do anyting...open the console and say &quot; press enter to continue&quot;... I press and...end... </p><p>can anybody help me please? :)</p>
<p>Hi!</p><p>Is it OK if I have this message after compiling? Thank you!</p><p>1&gt;------ Build started: Project: aver, Configuration: Debug Win32 ------<br>1&gt;aver.obj : warning LNK4075: ignoring '/EDITANDCONTINUE' due to '/INCREMENTAL:NO' specification<br>1&gt; aver.vcxproj -&gt; C:\Users\user\Documents\Visual Studio 2010\Projects\aver - copia\Debug\aver.exe<br>========== Build: 1 succeeded, 0 failed, 0 up-to-date, 0 skipped ==========</p>
<p>Hello, this is my first arduino project: sending a file (jpeg, pdf, txt)<br> from an arduino sd card module to android phone using arduino uno and <br>bluetooth module HC-05<br>It's possible with this hardware?<br><br>If it's possible can someone please send me the source code and android app use it to receive the file<br>thank you</p>
<p>Can anyone upload the full project folder? It will be helpful for me?</p>
<p>hi </p><p> i run your program with visual C++ 2010 express as you said but i have the following errors may i request you to help me </p><p>//---------------------------------------the error lis is below </p><p>1&gt;------ Build started: Project: projectbs01, Configuration: Debug Win32 ------<br>1&gt;techbitarFaceDetection.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::disconnect(void)&quot; (?disconnect@Tserial@@QAEXXZ) referenced in function _main<br>1&gt;techbitarFaceDetection.obj : error LNK2019: unresolved external symbol &quot;public: int __thiscall Tserial::connect(char *,int,enum serial_parity)&quot; (?connect@Tserial@@QAEHPADHW4serial_parity@@@Z) referenced in function _main<br>1&gt;techbitarFaceDetection.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::Tserial(void)&quot; (??0Tserial@@QAE@XZ) referenced in function _main<br>1&gt;techbitarFaceDetection.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::~Tserial(void)&quot; (??1Tserial@@QAE@XZ) referenced in function &quot;public: void * __thiscall Tserial::`scalar deleting destructor'(unsigned int)&quot; (??_GTserial@@QAEPAXI@Z)<br>1&gt;techbitarFaceDetection.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::sendChar(char)&quot; (?sendChar@Tserial@@QAEXD@Z) referenced in function &quot;void __cdecl detectAndDisplay(class cv::Mat)&quot; (?detectAndDisplay@@YAXVMat@cv@@@Z)<br>1&gt;C:\Users\Arash\Documents\Visual Studio 2010\Projects\projectbs01\Debug\projectbs01.exe : fatal error LNK1120: 5 unresolved externals<br>========== Build: 0 succeeded, 1 failed, 0 up-to-date, 0 skipped ==========</p>
<p>Solved.</p><p>just copy following text at top of your codes in c++.</p><p>#include&lt;tserial.cpp&gt;</p>
<p>To use this code with OpenCV3, you have to change out the way the program captures frames from the webcamera:<br><br>At the top of main, swap out &quot;CvCapture* capture;&quot; with &quot;VideoCapture capture(0);&quot;.<br>A few lines further down you find the capture attempt with: capture = cvCaptureFromCAM( 3 );, which can be removed entirely. While you're at it, remove the if sentence underneath it as well. It checks if it's able to capture with the camera, but is in reality completely pointless as if the camera isn't working you'll be getting an error before it. An error that you want, mind you, so trying to toss it aside with an if-sentence and exiting the program isn't exactly good code.<br>Next swap out the line &quot;frame = cvQueryFrame( capture );&quot; with &quot;capture &gt;&gt; frame;&quot; and you're all done.</p>
<p>Hi AndreasL8, thanks it help me with opencv 3.1. I need to change one line to make it work.</p><pre>&quot;if( capture )&quot; change with &quot;if( capture.isOpened() )&quot;</pre>
<p>I'm using a Windows 8.1 machine, running OpenCV 2.3.1 and Visual C++ Express 2010. When I compile the FE73Q6BGVXH5LPM program, I get some errors I can't figure out (I'm a C++ rookie, but am a programmer in other older languages). This is the first of two types:</p><p>1&gt;c:\users\matt\documents\visual studio 2010\projects\opencv test2\opencv test2\fe73q6bgvxh5lpm.cpp(59): error C2679: binary '=' : no operator found which takes a right-hand operand of type 'IplImage *' (or there is no acceptable conversion)</p><p>This is referring to the assignment, 'frame = cvQueryFrame( capture );' which references 'CVAPI(IplImage*) cvQueryFrame( CvCapture* capture );' out of the videoio_c.h* file. </p><p>I'm not having much luck figuring this one out. The description of this C2679 error code doesn't help much, being new to the language.</p><p>The second type (there are 4 similar to this) is. Not sure if they're fatal errors or warnings:</p><p>1&gt; c:\program files (x86)\microsoft visual studio 10.0\vc\include\opencv2\core\mat.hpp(948): could be 'cv::Mat &amp;cv::Mat::operator =(const cv::Mat &amp;)', which is referring to:</p><p>'Mat&amp; operator = (const Mat&amp; m);' in mat.hpp.</p><p>I suspect my setup and installation process may be at fault, but don't know. I did follow the instructions posted. But some further instruction or direction to go in researching this would be very much appreciated!</p><p>Matt</p>
<p>Can i use here codeblocks instead of visual studio ??</p>
<p>hi, can i do the same thing with an arduino ATmega168 ??</p>
<p>this tutorial arent useful unless you know how to install and run opencv in the pc using visual studios</p>
<p>Please help!!!. i could not correct my c2664 CreateFile() : cannot convert argument 1 from char[10] to 'LPCWSTR' problem on tserial.cpp code. Are there anyone fix this problem? Can anyone tell me exactly what should we modify in the tserial.cpp ?</p>
<p>I wonder if it is doable or Raspberry Pi 2 powered by Windows 10? </p><p>It would be nice setup having Arduino + Raspberry working jointly without involvement of massive PC</p>
<p>nice topic</p>
<p>Hi! I am a grade 9 student and i would like to ask if your detector knows who is in the picture. Does this device shows the personal information of the men who is in the picture? I would like to know the answer as much as possible.. thank you so much! </p>
<p>The face tracking works well, but im stuck on making the servo rotate. I cant figure it out why it doesnt rotate. I changed the COM port on my COM3, but still no result. Any help would be greatly appreciated. Thanks</p>
<p>can u please help me to make this because i want this project as my final project but still have no idea how to make please reply must</p>
<p>has exited with code -1 (0xffffffff). <br>anyone can help me?</p>
<p>You might be using old OpenCV version. This guide is based on 2.3.1. download it here </p><p>http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/OpenCV-2.3.1-win-superpack.exe/ and dont forget to change lib names/environmental variables</p>
<p>There is one thing you'd have to correct in the part of code where COM port is identificated: if (arduino_com-&gt;connect(&quot;\\\\.\\COM13&quot;, 57600, spNONE)!=0). Usually, arduino drivers organise virtual COM's with numbers more then 9. For correct OpenFile function working you should use \\\\.\\COM13 and not just COM13.</p>
<p>I try to make this Arduino Open CV tutorial for webcams. OpenCV 3.0 <br>RC1 is compiled and installed. I also use Code::Blocks 13.12 as IDE. I <br>am a newbie what comes to OpenCV and I do not know too much about <br>Arduino either but here I am learning. Arduino IDE is 1.7.2. This <br>tutorial is made for Windows 7 and software for this OS but I bet it is <br>possible to repeat this in Ubuntu Linux 14.04 also. </p><p>When I try to compile and run the techbitarFaceDetection.cpp </p><p>the Code::Block IDE gives me following error:</p><blockquote> <br> <br>fatal error: Tserial.h: No such file or directory</blockquote><p>So please tell me what to do next and how to fix this error. </p>
<p>So, I have a question.</p><p>Arduino is just rotating the stepper motor, right? All the processing is done via C++ serial port...It's not a portable project unless I use RPi...</p>
<p>Can anybody help me out with this problem?</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::disconnect(void)&quot; (?disconnect@Tserial@@QAEXXZ) referenced in function _main</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: int __thiscall Tserial::connect(char *,int,enum serial_parity)&quot; (?connect@Tserial@@QAEHPADHW4serial_parity@@@Z) referenced in function _main</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::Tserial(void)&quot; (??0Tserial@@QAE@XZ) referenced in function _main</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: __thiscall Tserial::~Tserial(void)&quot; (??1Tserial@@QAE@XZ) referenced in function &quot;public: void * __thiscall Tserial::`scalar deleting destructor'(unsigned int)&quot; (??_GTserial@@QAEPAXI@Z)</p><p>1&gt;facedetect.obj : error LNK2019: unresolved external symbol &quot;public: void __thiscall Tserial::sendChar(char)&quot; (?sendChar@Tserial@@QAEXD@Z) referenced in function &quot;void __cdecl detectAndDisplay(class cv::Mat)&quot; (?detectAndDisplay@@YAXVMat@cv@@@Z)</p>
<p>in the &quot;techbitarFaceDetection.cpp&quot; file you will include your Tserial files.</p><p>If you check, probably you will see that the only included file is Tserial.h</p><p>by adding #include &quot;Tserial.cpp&quot; you will solve this problem . This is how i solved mine</p>
<p>Thats so... CUTE :D</p>
<p>Hi, thank you so much for your helpful instructions. It really helps me. I am now at the step of <strong>OpenCV &amp; Microsoft Visual C++ Integration </strong>and I referred to the link OpenCV 2.3.1 and Visual Studio 2010</p><p><a href="http://www.deveature.com/2011/11/24/opencv-2-3-1-and-visual-studio-2010/" rel="nofollow">http://www.deveature.com/2011/11/24/opencv-2-3-1-and-visual-studio-2010/</a></p><p>I saw your comment there where you had problem with &ldquo;Cannot find or open the PDB file&rdquo;. I am now having the same problem. May I know how did you solve it?</p><p>Thank you so much for your help.</p>
<p>i fixed it using this solution - http://answers.opencv.org/question/12600/problem-in-debugging-cannot-find-or-open-the-pdb/</p>
<p>Thanks for very instructive steps. I have made it and it is so cool. </p>
Hello there...buddy i need to know some thing from u
<p>Hi,</p><p>I'm trying to do the project with Code Blocks instead of Visual Studio.</p><p>Like a lot of people, I have an error with tserial.h: &quot;undefined reference to Tserial::Tserial().</p><p>I've added the library to the path. Do you have any idea of the problem?</p><p>Thank you very much!! </p>
<p>Hi, in the arduino file`s &quot;#include&quot;(i`m new on this), in your code, i don`t see what`s included anywhere. Can someone help on this?</p><p>Thanks</p>
Hi, i don't know what i need to make serial communication between c++ and arduino. I download Serial C++ Library for Win32 (by Thierry Schneider), I found sertest2.cpp, tserial.cpp and tserial.h, but i don't know where i have to put or include them and i don't know if there is another configrations needed. please don't skimp on the advice.
put all of them in your project folder..
Thanks for the tutorial..After countless hours testing, i made it..pheww.. <br>anyway, to avoid error, don't forget to put Tserial.cpp and Tserial.h in your project folder..that should fix LNK2019 problem i guess..
hmm, bounding box issue? <br>
I have everything wired up, fixed the Tserial issue, fixed the LINK2019 and TBB_dll issues. Now I am gettign this error message. <br> <br>The program '[10936] techbitarFaceDetection.exe: Native' has exited with code -1073741701 (0xc000007b). <br> <br>I am stumped. Does anybody have any idea why this is happening? <br>
it would cause a little bit trouble if you are doing with 64-bit system. because C++ 2010 express could only compile with *64 compiler which is contained in Windows 7.1SDK. SDK always fail me.
Btw it for me the servo were run too slow.. not as fast as it is in the video. <br> <br>Anybody here know how to make the servos run smoothly?
As many other people, im having troubles with implementing the library Tserial, By Thierry Schneider. Can anyone please clarify how to make the library work?, as many of us, im getting the conversion to &quot;LPCWSTR&quot; error.
Where to make change for port and baud rate in Tserial.cpp Help me ...

About This Instructable




Bio: Did I unplug the solder iron?
More by techbitar:IR Remote Control Station for Android - TURN THE TV DOWN SensoDuino: Turn Your Android Phone into a Wireless Sensors Hub for Arduino Modify The HC-05 Bluetooth Module Defaults Using AT Commands 
Add instructable to: