Introduction: MATLAB Color Recognition
Introduction:
In present scenario home office appliance controlling is common thing. Today for controlling home/offices appliances, machinery in companies/industries uses different types of automation systems like DTMF controlled, GSM controlled and many others. So here this project is made for interfacing MATLAB (Graphic User Interfacing) and Home/office/industries automation using Arduino and MATLAB which can controlled by showing certain symbols / images in front of the webcam. like to switch on the fan you need to show fan symbol in front of the webcam and to switch on the light you need to show light symbol in front of the webcam.
Here MATLAB is interfaced with Arduino through serial communication. Once the MATLAB process the image in the video taken by the webcam, it will send certain serial commands to arduino then,the arduino will turn on / off the respective appliances.
Note: I bought some part of the code from inscrutables, long ago that's why i'm unable recollect their names, but the credit goes to them, as this is not a commercial one,i used their work without proper permission.
Project_files can be found here: (please go through next steps for better understanding)
Step 1: Led Control Using Image Processing
to make This project simple and effective, i followed these steps.
- i want to detect the red color object then.
- count the number of objects (red) in the real time video.
- then i write the code in such way that depending upon the count,mat lab will send the data (please look in the code attachment) to arduino.
- then write the code for serial read based appliance controlling for arduino.
that's it.
Please look @ the picture carefully, that will give you entire idea
Matlab code:
%Initialise video capturing object
vid = videoinput('winvideo', 1, 'YUY2_640x480');
%SERIAL COMUNICATION%
%Declaring the serial comunication object , to initialise serial %comunication
s=serial('COM9','BAUD',9600); %change COM_ if needed
%open serial port fopen(s);
%VIDEO PROPERTIES SETTNG UP% % Set the properties of the video object
set(vid, 'FramesPerTrigger', Inf); set(vid, 'ReturnedColorspace', 'rgb') vid.FrameGrabInterval = 10;
%start the video aquisition here
start(vid)
% Set a loop that stop after 100 frames of aquisition %for i=1:100
while(vid.FramesAcquired<=20000)
% Get the snapshot of the current frame
IMRED = getsnapshot(vid);
% Now to track red objects in real time%
% we have to subtractED the red component % from the grayscale image to extract the red components in the image.
diff_im = imsubtract(IMRED(:,:,1), rgb2gray(IMRED));
gr=graythresh(diff_im);
%Median filter IS USED to filter out noise
diff_im = medfilt2(diff_im, [3 3]);
% Convert the resulting grayscale image into a binary image.
diff_im = im2bw(diff_im,0.18); % Remove all those pixels less than 300px
diff_im = bwareaopen(diff_im,300); % Label all the connected components in the image
%and also count the nuber of red objects in frame
[bw bw1] = bwlabel(diff_im, 8);
% condition if one or more than one red object is present in frame,then %send value '100' else send 101
if bw1=1
fprintf(s,100); elseif bw1==0 fprintf(s,101);
if bw1=2
fprintf(s,110); elseif bw1==0 fprintf(s,111);
if bw1=3
fprintf(s,120); elseif bw1==0 fprintf(s,121);
% Here we do the image blob analysis.
% We get a set of properties for each labeled region.
stats = regionprops(bw, 'BoundingBox', 'Centroid');
% Display the image
imshow( IMRED )
hold on
%This is a loop to bound the red objects in a rectangular box.
for object = 1:length(stats)
bb = stats(object).BoundingBox;
bc = stats(object).Centroid;
rectangle('Position',bb,'EdgeColor','r','LineWidth',2)
plot(bc(1),bc(2), '-m+')
a=text(bc(1)+15,bc(2), strcat('X: ', num2str(round(bc(1))), ' Y: ', num2str(round(bc(2))))); set(a, 'FontName', 'Arial', 'FontWeight', 'bold', 'FontSize', 12, 'Color', 'yellow');
end
hold off
end
end
% Both the loops end her. % Stop the video aquisition.
stop(vid);
% Flush all the image data stored in the memory buffer.
flushdata(vid);
%Close the serial port ,so that further communication could be establihed %again.
fclose(s); % Clear all variables clear all
sprintf('%s','Congratulations')
Arduino code:
const int rpin=2;
const int gpin=5; const int bpin=8; int recValue;
void setup()
{
Serial.begin(9600);
pinMode(2, OUTPUT); pinMode(5, OUTPUT); pinMode(8, OUTPUT);
}
void loop()
{
if(Serial.available()>0)
{
recValue=Serial.read();
if (recValue == 100) // If use will send value 100 from MATLAB then r LED will turn ON
{ digitalWrite(rpin, HIGH); }
if(recValue == 101) // If use will send value 101 from MATLAB then r LED will turn OFF
{ digitalWrite(rpin, LOW); }
if (recValue == 110) // If use will send value 110 from MATLAB then gLED will turn ON
{ digitalWrite(gpin, HIGH); } if(recValue == 111) // If use will send value 111 from MATLAB then gLED will turn OFF
{ digitalWrite(gpin, LOW); } if (recValue == 120) // If use will send value 120 from MATLAB then bLED will turn ON
{ digitalWrite(bpin, HIGH); } if(recValue == 121) // If use will send value 121 from MATLAB then bLED will turn OFF
{ digitalWrite(bpin, LOW); } }
}
Step 2: Image Based Control
to make This project simple and effective, i followed these steps.
- i want to detect the red,green,blue colors object then.
- count the number of objects (red/green/blue) in the real time video.
- then i write the code in such way that depending upon the respective count >=1, mat lab will send the data (please look in the code attachment) to arduino
- .then write the code for serial read based appliance controlling for arduino. that's it.
Please look @ the picture carefully, that will give you entire idea
Matlab code:
redThresh = 0.20; % Threshold for red detection
greenThresh = 0.10; % Threshold for green detection
blueThresh = 0.15; % Threshold for blue detection
s=serial('COM8','BAUD',9600); %open serial port fopen(s);
vidDevice = imaq.VideoDevice('winvideo', 1, 'YUY2_640x480', ... % Acquire input video stream
'ROI', [1 1 640 480], ... 'ReturnedColorSpace', 'rgb');
vidInfo = imaqhwinfo(vidDevice); % Acquire input video property
hblob = vision.BlobAnalysis('AreaOutputPort', false, ... % Set blob analysis handling 'CentroidOutputPort', true, ... 'BoundingBoxOutputPort', true', ... 'MinimumBlobArea', 600, ... 'MaximumBlobArea', 3000, ... 'MaximumCount', 10);
hshapeinsBox = vision.ShapeInserter('BorderColorSource', 'Input port', ... % Set box handling 'Fill', true, ... 'FillColorSource', 'Input port', ... 'Opacity', 0.4);
htextinsRed = vision.TextInserter('Text', 'Red : %2d', ... % Set text for number of blobs 'Location', [5 2], ... 'Color', [1 0 0], ... // red color 'Font', 'Courier New', ... 'FontSize', 14);
htextinsGreen = vision.TextInserter('Text', 'Green : %2d', ... % Set text for number of blobs 'Location', [5 18], ... 'Color', [0 1 0], ... // green color 'Font', 'Courier New', ... 'FontSize', 14);
htextinsBlue = vision.TextInserter('Text', 'Blue : %2d', ... % Set text for number of blobs 'Location', [5 34], ... 'Color', [0 0 1], ... // blue color 'Font', 'Courier New', ... 'FontSize', 14);
htextinsCent = vision.TextInserter('Text', '+ X:%4d, Y:%4d', ... % set text for centroid 'LocationSource', 'Input port', ... 'Color', [1 1 0], ... // yellow color 'Font', 'Courier New', ... 'FontSize', 15);
hVideoIn = vision.VideoPlayer('Name', 'Final Video', ... % Output video player 'Position', [100 100 vidInfo.MaxWidth+20 vidInfo.MaxHeight+30]); nFrame = 0; % Frame number initialization
%% Processing Loop
while(nFrame < 3000)
rgbFrame = step(vidDevice); % Acquire single frame
rgbFrame = flipdim(rgbFrame,2); % obtain the mirror image for displaying
diffFrameRed = imsubtract(rgbFrame(:,:,1),
rgb2gray(rgbFrame)); % Get red component of the image
diffFrameRed = medfilt2(diffFrameRed, [3 3]); % Filter out the noise by using median filter
binFrameRed = im2bw(diffFrameRed, redThresh); % Convert the image into binary image with the red objects as white
%binFrameRed = bwareaopen(binFrameRed,300);
[bw bw1] = bwlabel(binFrameRed, 8);
if bw1>=1
fprintf(s,101);
else
fprintf(s,100);
end
diffFrameGreen = imsubtract(rgbFrame(:,:,2), rgb2gray(rgbFrame)); % Get green component of the image diffFrameGreen = medfilt2(diffFrameGreen, [3 3]); % Filter out the noise by using median filter
binFrameGreen = im2bw(diffFrameGreen, greenThresh); % Convert the image into binary image with the green objects as white
%binFrameRed = bwareaopen(binFrameGreen,300);
[bw bw1] = bwlabel(binFrameGreen, 8);
if bw1>=1
fprintf(s,102);
else
fprintf(s,103);
end
diffFrameBlue = imsubtract(rgbFrame(:,:,3), rgb2gray(rgbFrame)); % Get blue component of the image diffFrameBlue = medfilt2(diffFrameBlue, [3 3]); % Filter out the noise by using median filter
binFrameBlue = im2bw(diffFrameBlue, blueThresh); % Convert the image into binary image with the blue objects as white
%binFrameRed = bwareaopen(binFrameBlue,300);
[bw bw1] = bwlabel(binFrameBlue, 8);
if bw1>=1
fprintf(s,104);
else
fprintf(s,105);
end
[centroidRed, bboxRed] = step(hblob, binFrameRed); % Get the centroids and bounding boxes of the red blobs centroidRed = uint16(centroidRed); % Convert the centroids into Integer for further steps
[centroidGreen, bboxGreen] = step(hblob, binFrameGreen); % Get the centroids and bounding boxes of the green blobs centroidGreen = uint16(centroidGreen); % Convert the centroids into Integer for further steps [centroidBlue, bboxBlue] = step(hblob, binFrameBlue); % Get the centroids and bounding boxes of the blue blobs centroidBlue = uint16(centroidBlue); % Convert the centroids into Integer for further steps rgbFrame(1:50,1:90,:) = 0; % put a black region on the output stream
vidIn = step(hshapeinsBox, rgbFrame, bboxRed, single([1 0 0])); % Instert the red box
vidIn = step(hshapeinsBox, vidIn, bboxGreen, single([0 1 0])); % Instert the green box
vidIn = step(hshapeinsBox, vidIn, bboxBlue, single([0 0 1])); % Instert the blue box
for object = 1:1:length(bboxRed(:,1)) % Write the corresponding centroids for red
centXRed = centroidRed(object,1);
centYRed = centroidRed(object,2);
vidIn = step(htextinsCent, vidIn, [centXRed centYRed], [centXRed-6 centYRed-9]);
end
for object = 1:1:length(bboxGreen(:,1)) % Write the corresponding centroids for green
centXGreen = centroidGreen(object,1);
centYGreen = centroidGreen(object,2);
vidIn = step(htextinsCent, vidIn, [centXGreen centYGreen], [centXGreen-6 centYGreen-9]);
end
for object = 1:1:length(bboxBlue(:,1)) % Write the corresponding centroids for blue
centXBlue = centroidBlue(object,1); centYBlue = centroidBlue(object,2);
vidIn = step(htextinsCent, vidIn, [centXBlue centYBlue], [centXBlue-6 centYBlue-9]);
end
vidIn = step(htextinsRed, vidIn, uint8(length(bboxRed(:,1)))); % Count the number of red blobs
vidIn = step(htextinsGreen, vidIn, uint8(length(bboxGreen(:,1)))); % Count the number of green blobs
vidIn = step(htextinsBlue, vidIn, uint8(length(bboxBlue(:,1)))); % Count the number of blue blobs
step(hVideoIn, vidIn); % Output video stream
nFrame = nFrame+1;
end
%% Clearing Memory release(hVideoIn); % Release all memory and buffer used
release(vidDevice);
clear all;
clc;
Arduino code:
const int led1pin=2;
const int led2pin=3; const int led3pin=4; int recValue;
void setup()
{
Serial.begin(9600);
pinMode(2, OUTPUT); pinMode(3, OUTPUT); pinMode(4, OUTPUT); pinMode(5, OUTPUT);
}
void loop()
{
if(Serial.available()>0)
{
recValue=Serial.read();
if (recValue == 100) // If use will send value 100 from MATLAB then LED will turn ON
{ digitalWrite(led1pin, HIGH); }
if(recValue == 101) // If use will send value 101 from MATLAB then LED will turn OFF
{ digitalWrite(led1pin, LOW); } if (recValue == 102) // If use will send value 100 from MATLAB then LED will turn ON
{ digitalWrite(led2pin, HIGH); }
if(recValue == 103) // If use will send value 103 from MATLAB then LED will turn OFF
{ digitalWrite(led2pin, LOW); } if (recValue == 104) // If use will send value 100 from MATLAB then LED will turn ON
{ digitalWrite(led3pin, HIGH); }
if(recValue == 105) // If use will send value 105 from MATLAB then LED will turn OFF
{ digitalWrite(led3pin, LOW); }
}
}