Face Detection and Tracking With Arduino and OpenCV

433,605

633

100

About: Did I unplug the solder iron?

Intro: Face Detection and Tracking With Arduino and OpenCV

UPDATES
  • Feb 20, 2013: In response to a question by student Hala Abuhasna if you wish to use the .NET Serial class, use the naming convention "\\\\.\\COMn" and replace n with a number > 9 to define your com port for COM ports above 9 such as COM10, COM11, etc.  
  • Mar 23, 2012: Featured on Adafruit's Blog
  • Mar 23, 2012: Featured on Interactive Design
  • Dec 13, 2011: Featured on Floss For Science

This guide will also be maintained on my blog http://techbitar.blogspot.com/2012/04/face-detection-and-tracking-with.html

INTRODUCTION

In this project I have assembled a face detection and tracking system. You can see the video of the final project here:



Basically, the webcam sends video frames to OpenCV running on a Windows PC. If OpenCV detects a face it will track it and calculate its center's X,Y coordinates. The coordinates are then passed on to the Arduino via a serial USB connection. The Arduino controls the movement of the webcam with the help of two pan/tilt servos to follow the detected face.

OpenCV (Open Source Computer Vision Library: http://opencv.willowgarage.com/wiki/) is an open-source library that includes several hundreds of real-time computer vision algorithms. The OpenCV 2.x library is a C++ API.

This is an integration project between hardware and software tools. The image processing C++ code samples are provided with the openCV library and all I did was to modify the sample code for this project. I removed some of the unnecessary code and added serial communications to it so it can send X,Y values to Arduino.

CREDIT

This project would not have been possible without the team who developed OpenCV. I also benefited from Ryan Owens' tutorial (http://www.sparkfun.com/tutorials/304) which is based on Processing, OpenFrameworks, and an earlier version of OpenCV. I tried to install Processing and OpenFrameworks with no luck. So Instead, I opted for Microsoft Visual C++ 2010 Express and the latest version of OpenCV which is 2.3.1 without any middleware or wrappers.

TOOLS

Software Required

Arduino IDE 1.0 for Windows
OpenCV 2.3.1 SuperPack For Windows
Microsoft Visual C++ 2010 Express SP1
Serial C++ Library for Win32 (by Thierry Schneider)

Code Required

- OpenCV C++ (attached) techbitarFaceDetection.cpp (based on OpenCV's example facedetect.cpp)
- Arduino's (attached) cam_servo.ino (based on Ryan Owens' example SerialServoControl.pde)

Hardware Required

- PC preferably running Windows 7 SP1. The faster CPU the better.
- Arduino Uno or compatible + power source.
- Standard servos X 2.
- Webcam w/UBS interface.
- Breadboard.
- Jumper wires.
- Hobby wire to tie pan/tilt servos and webcam together.

   

Step 1: Installation and Integration Issues

1) Download and install the OpenCV-2.3.1-win-superpack.exe if you don't wish to deal with generating the support files yourself. Everything you need from OpenCV to build this project has already been generated in this download.

http://sourceforge.net/projects/opencvlibrary/files/opencv-win/2.3.1/

2) Download and install Microsoft Visual C++ 2010 Express

http://www.microsoft.com/visualstudio/en-us/products/2010-editions/visual-cpp-express

The OpenCV installation documentation explains how to make Visual C++ aware of the OpenCV support files (include, bin, etc). This is not a one-click job. Careful attention must be given to how Visual C++ must be configured to recognize OpenCV files.

The OpenCV team tested version 2.3.1 and Visual C++ 2010 on Windows 7 SP1.  If you are using a different configuration, be prepared for a few hiccups.

Step 2: Attach Servos and Camera

I did not want to affix any of the project parts permanently because I like to take my projects apart after I am done.  So I used hobby wire, which is nothing more than a stiff wire, to tie the servos and the webcam together.

I wrapped the base of the webcam to the pan servo horn. Then I wrapped a cable around the horn of the tilt servo and the body of the pan servo.

To keep the whole servo/webcam assembly from moving randomly during operation, I used a clamp to tie it down to the soldering helping hands. It kept is steady during servo rotation.

It ain't pretty but it works.

Step 3: Wiring the Parts Together

The wiring is straight forward. I used a breadboard to make the connections.

SERVOS
The yellow/signal wire for the pan (X axis) servo goes to digital pin 9.
The yellow/signal wire for the tilt (y axis) servo goes to digital pin 10.
The red/Vcc wires of both servos go to the Arduino's 5V pin.
The black/GND wires of both servos go to Arduino's GND pin.

WEBCAM
The webcam's USB goes to the PC. The C++ code will identify it via a number representing the USB port its connected to. .

ARDUINO
The Arduino Uno is connected to the PC via USB. Take note of the COM port the USB is connected to. You can find the com port from the Arduino Tools/Serial Ports menu. You will see a check mark next to the active USB port. This is the com port that you will use in your C++ code to communicate with Arduino.

You must change the C++ code to match the PC's com port and baud rate with Arduino's. Also, the C++ code must be told which USB port the webcam is using.

Step 4: Resources

I found these websites to be informative:

OpenCV Face Detection/Tracking Guides

Face Tracking with a Pan/Tilt Servo Bracket by zagGrad
http://www.sparkfun.com/tutorials/304

Arduino + Servo + openCV Tutorial [openFrameworks]
http://www.creativeapplications.net/tutorials/arduino-servo-opencv-tutorial-openframeworks


OpenCV & Microsoft Visual C++ Integration

OpenCV 2.3.1 and Visual Studio 2010
http://www.deveature.com/2011/11/24/opencv-2-3-1-and-visual-studio-2010/

Getting Started with OpenCV 2.3 in Microsoft Visual Studio 2010 in Windows
http://siddhantahuja.wordpress.com/2011/07/18/getting-started-with-opencv-2-3-in-microsoft-visual-studio-2010-in-windows-7-64-bit

OpenCV 2.1.0 with Visual Studio 2010
http://blog.aguskurniawan.net/post/OpenCV-210-with-Visual-Studio-2010.aspx

How to build applications with OpenCV inside the Microsoft Visual Studio
http://opencv.itseez.com/doc/tutorials/introduction/windows_visual_studio_Opencv/windows_visual_studio_Opencv.html

Using OpenCV 2.3.1 with Visual Studio 2010 (tutorial)
http://www.anlak.com/using-opencv-2-3-1-with-visual-studio-2010-tutorial

Share

    Recommendations

    • Audio Contest 2018

      Audio Contest 2018
    • Plastics Contest

      Plastics Contest
    • Optics Contest

      Optics Contest

    100 Discussions

    0
    None
    anuj_js

    2 months ago

    nice

    0
    None
    rohanday4

    Question 6 months ago

    can you program it to recognise faces and take pictures as well?

    0
    None
    ahmed.siam1250

    7 months ago

    i am need this but buy using Python

    0
    None
    Allie Quinn

    Question 7 months ago

    Hello! I got this error and I don't know how to resolve it:

    Error1error C2679: binary '=' : no operator found which takes a right-hand operand of type 'IplImage *' (or there is no acceptable conversion)d:\users\naseef\documents\visual studio 2008\projects\thesistrial1\thesistrial1\thesisfirsttrial.c58ThesisTrial1

    0
    None
    somil.nath

    8 months ago

    i got the below error can anyone tel me how to resolve this error:
    Error (active) no operator "=" matches these operands
    c:\Users\SPJ\Documents\Visual Studio 2015\Projects\ConsoleApplication2\ConsoleApplication2\Source.cpp types are: cv::Mat = IplImage *

    1 reply
    0
    None
    XhensilaP1

    9 months ago

    i am using windows 7 professional, is it ok not to use Windows SP1 for this configurations? Please i need an answer as soon as posible. thanks in advance.

    0
    None
    Gaurav64

    1 year ago

    hi I wan to write a code in general to detect any object of any size is it possible?

    1 reply
    0
    None
    Smees69Gaurav64

    Reply 1 year ago

    Wouldn't that just be a camcorder set to record movement?

    0
    None
    aykuts4

    1 year ago

    this tutorial is not working with visual studio 2017 so would you make a tutorial with lastest version of visual studio please ( not with visual studio 2010, it is too old ) ? tyvm

    1 reply
    0
    None
    Smees69aykuts4

    Reply 1 year ago

    Why don't you just update the code for 2017?

    0
    None
    Raknco28

    1 year ago

    Thak you.

    0
    None
    biswajit11

    1 year ago

    Is Logitech webcam can be used instead of Creative webcam? If yes, suggest the part number

    0
    None
    paoloa2l

    1 year ago

    Help

    1>------ Inizio compilazione: Progetto: RiconosceFaccia, Configurazione: Debug Win32 ------

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "void __cdecl cv::fastFree(void *)" (?fastFree@cv@@YAXPAX@Z) non risolto nella funzione "public: __thiscall cv::Mat::~Mat(void)" (??1Mat@cv@@QAE@XZ)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: __thiscall cv::_InputArray::_InputArray(class cv::Mat const &)" (??0_InputArray@cv@@QAE@ABVMat@1@@Z) non risolto nella funzione "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: __thiscall cv::_OutputArray::_OutputArray(class cv::Mat &)" (??0_OutputArray@cv@@QAE@AAVMat@1@@Z) non risolto nella funzione "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: __thiscall cv::Mat::Mat(class cv::Mat const &,class cv::Rect_<int> const &)" (??0Mat@cv@@QAE@ABV01@ABV?$Rect_@H@1@@Z) non risolto nella funzione "public: class cv::Mat __thiscall cv::Mat::operator()(class cv::Rect_<int> const &)const " (??RMat@cv@@QBE?AV01@ABV?$Rect_@H@1@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: __thiscall cv::Mat::Mat(struct _IplImage const *,bool)" (??0Mat@cv@@QAE@PBU_IplImage@@_N@Z) non risolto nella funzione _main

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: void __thiscall cv::Mat::deallocate(void)" (?deallocate@Mat@cv@@QAEXXZ) non risolto nella funzione "public: void __thiscall cv::Mat::release(void)" (?release@Mat@cv@@QAEXXZ)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: void __thiscall cv::Mat::copySize(class cv::Mat const &)" (?copySize@Mat@cv@@QAEXABV12@@Z) non risolto nella funzione "public: __thiscall cv::Mat::Mat(class cv::Mat const &)" (??0Mat@cv@@QAE@ABV01@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "void __cdecl cv::ellipse(class cv::Mat &,class cv::Point_<int>,class cv::Size_<int>,double,double,double,class cv::Scalar_<double> const &,int,int,int)" (?ellipse@cv@@YAXAAVMat@1@V?$Point_@H@1@V?$Size_@H@1@NNNABV?$Scalar_@N@1@HHH@Z) non risolto nella funzione "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: __thiscall cv::CascadeClassifier::CascadeClassifier(void)" (??0CascadeClassifier@cv@@QAE@XZ) non risolto nella funzione "void __cdecl `dynamic initializer for 'eyes_cascade''(void)" (??__Eeyes_cascade@@YAXXZ)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: virtual __thiscall cv::CascadeClassifier::~CascadeClassifier(void)" (??1CascadeClassifier@cv@@UAE@XZ) non risolto nella funzione "void __cdecl `dynamic atexit destructor for 'eyes_cascade''(void)" (??__Feyes_cascade@@YAXXZ)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: bool __thiscall cv::CascadeClassifier::load(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &)" (?load@CascadeClassifier@cv@@QAE_NABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@@Z) non risolto nella funzione _main

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "public: virtual void __thiscall cv::CascadeClassifier::detectMultiScale(class cv::Mat const &,class std::vector<class cv::Rect_<int>,class std::allocator<class cv::Rect_<int> > > &,double,int,int,class cv::Size_<int>,class cv::Size_<int>)" (?detectMultiScale@CascadeClassifier@cv@@UAEXABVMat@2@AAV?$vector@V?$Rect_@H@cv@@V?$allocator@V?$Rect_@H@cv@@@std@@@std@@NHHV?$Size_@H@2@2@Z) non risolto nella funzione "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno _cvCreateCameraCapture non risolto nella funzione _main

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno _cvQueryFrame non risolto nella funzione _main

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "void __cdecl cv::imshow(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class cv::_InputArray const &)" (?imshow@cv@@YAXABV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@ABV_InputArray@1@@Z) non risolto nella funzione "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "int __cdecl cv::waitKey(int)" (?waitKey@cv@@YAHH@Z) non risolto nella funzione _main

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "void __cdecl cv::equalizeHist(class cv::_InputArray const &,class cv::_OutputArray const &)" (?equalizeHist@cv@@YAXABV_InputArray@1@ABV_OutputArray@1@@Z) non risolto nella funzione "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

    1>RiconosceFaccia.obj : error LNK2019: riferimento al simbolo esterno "void __cdecl cv::cvtColor(class cv::_InputArray const &,class cv::_OutputArray const &,int,int)" (?cvtColor@cv@@YAXABV_InputArray@1@ABV_OutputArray@1@HH@Z) non risolto nella funzione "void __cdecl detectAndDisplay(class cv::Mat)" (?detectAndDisplay@@YAXVMat@cv@@@Z)

    1>C:\Users\Paolo3507\documents\visual studio 2017\Projects\RiconosceFaccia\Debug\RiconosceFaccia.exe : fatal error LNK1120: 18 esterni non risolti

    1>Compilazione progetto "RiconosceFaccia.vcxproj" NON COMPLETATA.

    ========== Compilazione: 0 completate, 1 non riuscite, 0 aggiornate, 0 ignorate ==========

    0
    None
    valway

    1 year ago

    Hello everyone!

    Someone was able to correct the error: can not convert argument 1 from char [10] to 'LPCWSTR'?

    Help me please!

    error.jpg
    0
    None
    tchapas

    1 year ago

    hi gooooooooooood jobbb....but i like to do this with a cart raspberry pi 2 and

    step-by-step engine......plzzzz samoane help me :( :( :(

    0
    None
    funkykomodo

    5 years ago on Introduction

    Yow all if you have problem with TSerial 'LPCWSTR' Error...
    See this link.. http://stackoverflow.com/questions/3924926/cannot-convert-parameter-1-from-char-to-lpcwstr

    I got it fixed :)

    2 replies
    0
    None
    maininlacifunkykomodo

    Reply 2 years ago

    Hi Funkykomodo,

    Although i visited the link that u wrote, i could not correct my c2664 CreateFile : cannot convert argument 1 from char[10] to 'LPCWSTR' problem on tserial.cpp code. Can you tell me exactly what you modified in the tserial.cpp ?

    0
    None
    BretN1maininlaci

    Reply 1 year ago

    I commented out

    //#include <stdafx.h>

    and then i changed

    serial_handle = CreateFile(port, GENERIC_READ | GENERIC_WRITE,

    0, NULL, OPEN_EXISTING,NULL,NULL);

    to

    serial_handle = CreateFileA(port, GENERIC_READ | GENERIC_WRITE,

    0, NULL, OPEN_EXISTING,NULL,NULL);

    Unfortunately, nothing is happening on my serial monitor yet

    0
    None
    sentiner

    1 year ago

    Hi

    I want to make a cam to recognize things by shape and color of it without using PC or computer (just arduino).

    please give me a URL or something to help.

    thanks