Introduction: Real Time Face Tracking Robot With Arduino and Matlab

Real time face tracking refers to the task of locating human faces in a video stream and tracking the detected or recognized faces.Nowadays , there are many real world applications of face detection and other image processing techniques.There are many algorithms like Viola Jones , Kanade-Lucas-Tomasi ,camshift etc. that are quite helpful in face detection.

In our project , we made use of some functions (in Matlab) that together make up the Viola-Jones face detection algorithm.We built a robot that tracks the movement of a face from the video stream using Matlab and Arduino.

Step 1: Components Used

  1. Arduino Uno Microcontroller.
  2. Matlab software(R2012b , installed with arduino I/O support package).

  3. Arduino IDE(1.0.6)

  4. Servo Motors x 2.

  5. Usb Webcam (Logitech 720p).

  6. Jumper cables.

Step 2: Working

Codes could be found attached in this step.

  1. Firstly , code in Matlab detects a face from every frame of the live video stream and inserts a bounding box around the Region of Interest., which is a face in this case(by detecting some haar features present in the human faces).The project code follows the Viola Jones algorithm for face detection.
  2. The set of frames with bounding boxes make up the addition of a bounding box around the face in live video.While adding a bounding box , we also calculate the coordinates of centroid of the bounding box.

  3. These coordinates are sent as a string to the arduino UNO microcontroller., from Matlab and these are processed according to the code written on arduino IDE for the movement of motors.

  4. During processing , the arduino gets the positions of PAN and TILT servo motors (that are attached as shown in the project image).Then , arduino checks if the centroid coordinates lie in the centre region of the screen. We are trying to move the camera in such a way that the centroid lies at the centre of the frame.(The pan and tilt servos are given to the digital pins 9 and 10 respectively...)

  5. For this reason the frame is divided into left and right halves and also top and bottom halves.If the centroid falls in the left half , the camera is panned right and if it falls in the right half , camera is panned left and the same with the top and bottom halves and tilting.

Step 3: References and Team Members

Comments

author
instructable123456789 made it! (author)2017-07-22

nice man so you should make is so the robot can move so it can follow you its like an obstacle avoiding car but following the human

author
Deik made it! (author)2017-07-16

Hey my matlab code is giving me error at the fopen function.always.can.u solve this please

author

Check if the arduino support package was installed properly in your matlab. we just used what we got in the case of cam and servos. Mostly, it should work with any type unless their functioning is different. I suggest try running some test code to control the servos using matlab once the support package is properly installed.

author
Deik made it! (author)2017-07-16

And there are specific camera and serxvo to make it work or it works in any type

author
HossamT3 made it! (author)2016-10-15

fopen(arduino);

this line gives me error

what do you think?

author
Thuta Zaw made it! (author)Thuta Zaw2017-05-04

same

author
DoDzilla made it! (author)2017-04-19

I am new to arduino. I've tried to copy your project. I have used the codes that you gave but for some reason when I run the matlab code, camera goes upward until my face dissappears from its field of view and then stops. Any suggestions? (maybe he doesnt like me... I need a plastic surgery...)

author
DoDzilla made it! (author)DoDzilla2017-04-19

and sometimes it gives this error in matlab during the test:

Warning: Out of range or non-integer values truncated during conversion to character.

> In camm2 (line 46)

and the line 46 is this :

fprintf(arduino,'%s',char(centy));

author
rhishabh_raunak made it! (author)2017-03-04

Nice work buddy...

Is the USB cable attached to arduino be replaced by a battery ? Is it used for power supply only or if not how can I make it laptop free ?

author
madhurya1595 made it! (author)2016-11-11

I'm getting this error:

Failed to open serial port COM4 to communicate with board Uno. Make sure there is no other MATLAB arduino object for this

board. For troubleshooting, see Arduino Hardware Troubleshooting.

But then I have updated the drivers for the ports,installed all arduino-matlab packages etc..I have connected only arduino to COM4.

I'm using Matlab 2016a. Kindly help.

author
madhurya1595 made it! (author)2016-10-20

How did you connect the 2 servo motors and the camera?Could you please explain the hardware set-up?

author

As you can notice in the block diagram or in the project photo itself, the tilt servo(y-control) is taped to the base. And to the wheel of this tilt motor, pan-servo(x-control) is taped. Finally, on the wheel of this pan-servo, cam is attached.

author
madhurya1595 made it! (author)madhurya15952016-10-23

Which servo motors are you using? I am trying to make it using Tower Pro MG995 servo motors,but not getting how to connect the two.Kindly help.

author
IbrahimM28 made it! (author)2016-05-18

hello

i'm trying to detect and track a red laser through matlab and arduino and i wrote the matlab code for detect the red object and i can get it's centroid but the problem is the connection between matlab and arduino and i cant let the servo motors track the red laser

any help please

thank you

author

Well I suggest that you test the motors through the Matlab first ,say send some arbitrary values to arduino and test whether it is working or not.There are two things u need to verify first: 1.check if the values are actually being sent from matlab to arduino or not.2. Check if u could control the motors using arduino or not....that's it ,the rest is all about the code.
If there is any problem in the 1st part, then possibly ur matlab's support package was not properly installed.
If the problem is in the 2nd part, then the problem might be either ur code or the arduino's serial connection.plz check all these things first.

author
IbrahimM28 made it! (author)2016-05-18

hello

i'm trying to detect and track a red laser through matlab and arduino and i wrote the matlab code for detect the red object and i can get it's centroid but the problem is the connection between matlab and arduino and i cant let the servo motors track the red laser

any help please

thank you

author
sharnam220 made it! (author)2016-04-29

Hello i am using this code to
control motor using arduino and matlab. But after running this program
output is always stop moving even if i move the ball in all directions.
Please help on this. Thanks :)

clear;
clc % Clearing Matlab desktop
vid=videoinput('winvideo',1,'YUY2_320x240'); % Defining the video input object
set(vid,'FramesPerTrigger',1); % Setting frames per trigger
preview(vid); %////// Showing the video of the moving Ball(TO BE USED %
% WHILE TESTING)
pause(10);% Waiting for a certain time for the system to get initialised
rgb_image = getsnapshot(vid); % Storing Image in an array variable
[a b c]= size(rgb_image); % Determining the size of the captured frame.
y=a;
x=b;


% Defining Boundaries
x1=x/2-120;
x2=x/2+120;
y1=y/2-30;
y2=y/2+30;
ser=serial('COM34'); % Defining the specified COM Port to be used
fopen(ser); % starting serial Communication,opening serial port

while(1)
rgb_image = getsnapshot(vid); % storing image in an array variable
flushdata(vid); %Flushing the buffer
rbar=0;
cbar=0;
e=0;
fR=rgb_image(:,:,1);fG=rgb_image(:,:,2);fB=rgb_image(:,:,3);% Storing RGB components of the image in seperate arrays
I=((fR<=70)
& (fG>=80) & (fB<=70)); % Converting the RGB Image into
binary image///Detecting only the red component
% Following are the steps For Detecting the red ball
se=strel('disk',20);
B=imopen(I,se);
final=imclose(B,se);
[L,n]=bwlabel(final);
%imshow(rgb_image); %////THIS IS TO BE USED ONLY WHILE TESTING
%hold on % ////THIS IS TO BE USED ONLY WHILE TESTING
for k=1:n
[r,c]=find(L==k);
rbar=mean(r);
cbar=mean(c);
%plot(cbar,rbar,'Marker','*','MarkerEdgeColor','B' ,'MarkerSize',20) %////THIS IS TO BE USED ONLY WHILE TESTING
e=(((cbar>=x1)*2*2*2) + ((cbar<=x2)*2*2) + ((rbar>=y1)*2) + (rbar<=y2)) % Converting to decimal number
end
% Decision Making Conditions
switch (e)
case 5
disp('Move left'),fprintf(ser,'L');
case 6
disp('Move left'),fprintf(ser,'L');
case 7
disp('Move left'),fprintf(ser,'L');
case 9
disp('Move right'),fprintf(ser,'R');
case 10
disp('Move right'),fprintf(ser,'R');
case 11
disp('Move right'),fprintf(ser,'R');
case 13
disp('Move forward'),fprintf(ser,'F');
case 14
disp('Move back'),fprintf(ser,'B');
otherwise
disp('Stop Moving'),fprintf(ser,'S');
end
end
fclose(ser); % closing serial port

author

Hello there...
I think its because the value of 'e' was something else than those specified in the cases.If I'm not wrong u r trying to move the camera (or possibly some other action through arduino)., based on the movement of the ball.There might be some problem in the calculation of 'e'. Try checking the value of 'e' every time before getting into the conditions.

author
shailendra60 made it! (author)2016-04-02

hello... i m new to this matlab.. so the file matlabcode.txt you have mentioned in the above 3 steps are only for the interfacing of matlab with arduino... can u plzz mail the complete code for this project at shankaj60@gmail.com

author

Hello...sorry for the late reply, the code in 'matlabcode.txt' is for interfacing matlab with arduino and image-processing as well. The rest is the 'arduino code'(Arduinocode.txt) for controlling the movement of the camera based on the image processing results.That's all the code...nothing else.

author
tato312 made it! (author)2016-04-21

Hi, first of all, thanks for the great tutorial, it's really helpful! I'm getting an error though, it's in the imaq.VideoDevice, it says the following:

Error using imaq.VideoDevice (line 270)
Invalid ADAPTORNAME 'winvideo' specified. ADAPTORNAME must be specified as a string. The available
adaptors are {(none)}. Image acquisition adaptors may be available as downloadable support
packages. Open Support Package Installer to install additional vendors.
Error in MatlabVideo (line 14)
obj =imaq.VideoDevice('winvideo', 1, 'I420_320x240','ROI', [1 1 320 240]);

I've already downloaded the support package for USB Webcams (and the Arduino I/O as well). I don't know if I'm missing something, or if I should change something depending on the configuration of my camera.

Can you help me?

Thanks in advance

author
tato312 made it! (author)tato3122016-04-21

Nevermind, I got it. If anyone has the same problem, all you have to do is download the OS Generic Video Interface from the Add-Ons tab in Matlab. Working now.

author
BhanuT4 made it! (author)2016-02-26

I am getting no serial port available error help me

author

try after going through the following link (It may help you) :

http://stackoverflow.com/questions/13224360/arduino-com-port-doesnt-work

author
BiswasL made it! (author)2016-02-17

Hello Friend,
I am able to track a color/face using Matlab in real time and identified its coordinate of centroid as well. I am using a ip camera for taking the video. Furthermore, i have also installed the matlab support package for Arduino. But i am struggling to go further.
I have read your post and happy to get some idea from this. as you said i think if i am able to send the coordicate of centroid only in the x axis to control the robot my problem would be solved. becuse my purpose to move the system only in the forward and reverse direction.
So i would like to request you to give step by step instruciton from communicating with arduino to control the system. that would be AWOSOME !!!!!. I am using the DC motor and L298N controller.
you can also sent code at lohanibiswas@yahoo.com.
Thanking you in advance.

author

Hello Biswas,....... so, u just want to make the camera pan (rotate sidewards..)., without tilting. Then , it is simple.Just comment all the lines that have to do with the tilting(y - co-ordinate)., both in the arduino and matlab.

author
BiswasL made it! (author)BiswasL2016-02-17

Hello Krishna, first of all thank you for sending the requested code. yes I just want to rotate in the side ward or like I am building the 4WD robot so want to move it only in the forward and reverse direction. if you give some idea how the control algorithm works that would be perfect !!!!

author
ayse_yilmaz made it! (author)2016-01-16

I use 2014a matlab. I didn't i/o for matlab. Shoul I use matlab 2012b ?
May you please help ?

author
bharath01028 made it! (author)bharath010282016-01-19

You can use both 2012b and 2014a.The one which we have done is on 2012b as know and we found I/O package for it .But if you use 2015b you need not have any special package .

author
ayse_yilmaz made it! (author)ayse_yilmaz2016-01-21

Do you have any tutorial for adding I/O package ?

author
bharath01028 made it! (author)bharath010282016-01-24

You Mentioned that you are using Matlab 2014a Version check whether there is an option 'addons' in the matlab home-page and if you have it then follow the below link to get the package installation

https://www.youtube.com/watch?v=3suGaVlyOOM

author
bharath01028 made it! (author)bharath010282016-01-24

You Mentioned that you are using Matlab 2014a Version check whether there is an option 'addons' in the matlab home-page and if you have it then follow the below link to get the package installation

https://www.youtube.com/watch?v=3suGaVlyOOM

author
bharath01028 made it! (author)bharath010282016-01-24

You Mentioned that you are using Matlab 2014a Version check whether there is an option 'addons' in the matlab home-page and if you have it then follow the below link to get the package installation

https://www.youtube.com/watch?v=3suGaVlyOOM

author
bharath01028 made it! (author)bharath010282016-01-24

You Mentioned that you are using Matlab 2014a Version check whether there is an option 'addons' in the matlab home-page and if you have it then follow the below link to get the package installation

https://www.youtube.com/watch?v=3suGaVlyOOM

author
bharath01028 made it! (author)bharath010282016-01-24

You Mentioned that you are using Matlab 2014a Version check whether there is an option 'addons' in the matlab home-page and if you have it then follow the below link to get the package installation

https://www.youtube.com/watch?v=3suGaVlyOOM

author

2014a is higher version and this should even work in it. u may encounter different steps for i/o support package installation.Any way, for i/o installation , try the commands 'targetinstaller' or 'supportpackageinstaller' and proceed.u can also find add-ons options for this purpose in the environment section in matlab.

author
ayse_yilmaz made it! (author)ayse_yilmaz2016-01-21

Do you have any tutorial for adding I/O package ?

author
Harika28 made it! (author)Harika282016-01-21

once try this link this document will probably guide you to install i/o package on your matlab .

http://in.mathworks.com/help/supportpkg/arduinoio/ug/intro.html

About This Instructable

16,897views

67favorites

License:

Bio: Currently ,studying at IIIT-SriCity, (India) and doing computer science (b.tech).
More by krishnachaithanya.movva:Real Time Face Tracking Robot With Arduino and Matlab
Add instructable to: