Begging Robot With Facial Tracking and Controll by Xbox Controller - Arduino

We are going to make a begging robot. This robot will try to irritate or get attention of passing people. It will detect their faces and try to shoot lasers at them. If you give the robot a coin, he will sing a song and dance. The robot will need a arduino, a live feed of a camera and a computer to run openCV on. The robot will also be able to be controlled by a xBox controller if connected to the PC.

Step 1: The Materials

Hardware Electronic

• Arduino NANO or UNO
• USB 2.0 Camera
• Jumper cables (male and female)
• 2 x Servo - Generic (Sub-Micro Size)
• 2 x LED - RGB CATHODE 5mm
• 2 x 5mW Lasers
• 1 x Red LED 5mm
• 1 x Breadboard
• 4 x 220Ω resistor
• 1 x 1KΩ resistor
• 1 x protoboard
• 1 x Sonar sensor 4 pins
• Xbox controller

Hardware analog

• Wooden box (15 x 15 x 7 cm)
• Glue
• Electrical tape

Software

• Arduino IDE
• Visual studio 2017
• 3Ds Max (or any other 3d moddeling software)
• Preform 2.14.0 or later
• OpenCV 3.4.0 or later

Tools

• Solder equipment
• Saw and drill
• Wire cutter

Step 2: Installing and Configuring OpenCV and C++

Step 2.1: Getting the software

Visual studio 2017: Download Visual studio Comunity 2017
openCV 3.4.0 Win pack: Go to official downloadpage

Step 2.2: Installing OpenCV
2.2.1: Extract the zipfile on to your Windows(:C) drive.
2.2.2: Go to your advanced system settings. This can be found in your win10 search function.
2.2.3: We need to setup new Environmental variables. Locate the "Path" environmental and press edit.
2.2.4: Now we have to add the location of the "bin map" to a new variable in the Path enviromental. If you installed openCV on you C drive, the path can go like this: C:\opencv\build\x64\vc14\bin Paste the path and press "OK" on all the windows that you may have opened during this proces.

Step 2.3: configuring visual studio C++
2.3.1: Make a new visual C++ project. Make it a empty win32 Console Application project.
2.3.2: In the source files tab, right click and add a new C++ file (.cpp) and name it "main.cpp".
2.3.3: Right click on the project-name in the Solution explorer and select Properties.
2.3.4: We need to add a additional Include Directories. This can be found under the C/C++ tab in general.
Copy the following path: C:\opencv\build\include and paste it behind the "AID" and click on apply.
2.3.5: In the same window we need to select the "Linker" tab. under general we need to make an other Additional Libary Directories. Paste tthe following path behind the "AID" C:\opencv\build\x64\vc14\lib and press apply again.
2.3.6: Under the same Linker tab select the "Input" tab. And press "Additional Dependencies > edit" and paste the following file opencv_world320d.lib and xinput.lib(For the controller) and press apply again. Close the window.
Now your C++ file is ready to work with.

Step 3: Setting Up the Arduino

Meeting the servos:
The servos are capible of rotating ~160°
They need to have between 4,8 and 6,0 Volt to operate normally.
The servo has 3 pins: ground, 4,8 - 6,0 volt pin and a data pin.
For our project we are going to set the data pins for the servos on DigitalPin 9 and 10.

Meeting the RGB leds:
The RGB leds have 4 pins.
Red, green, blue and ground pin.
In order to save some space on the arduino, we can connect the 2 RGB leds together. So we only will use 3 pins.
We can connect and solder the RGB leds on a protoboard like in the image.
Red pin => DigitalPin 3(PWM)
Green pin => DigitalPin 4
Blue pin => DigitalPin 7

Meeting the Piezo buzzer:
Our little robot is going to make some noise. In order to do this we need to give him a voice! We can choose to make him really loud. Or we can put a 220Ω resistor before the piezo buzzer to make him a little less obnoxious. We are leaving the Piezo buzzer on the breadboard. So no need for soldering.
We connect the data pin(+) to DigitalPin 2 and the ground pin to ground on the breadboard.

Meeting the Sonar:
In order to keep the robot from trying to aim at person that is 10 meters away. We can give the robot a distance range from where it will be able to aim at people. We do this with a sonar sensor.
VCC => 5 volt
Trig => DigitalPin 6
Echo => DigitalPin 5
GND => ground

Meeting the coin detector:
We are going to make a coin detector. The coin detector will work by detecting if the circuit is closed or broken. It will almost work like a switch. But we need to be carefull. If we do this wrong, it will cost us a arduino.
First: Connect AnalogPin A0 to a 5 volt cable. But make sure to put a 1KΩ resistor between it.
Second: Connect a wire to ground. We can immediately solder the wires and resistor to the same protoboard as the RGB leds.
Now if we touch the 2 wires toghter the arduino will detect a closed circuit. This means that there is a coin!

Meeting the lasers of doom.
The robot needs it's weapons to fire! To save some space, I solderd the 2 lasers together.
They will fit perfectly in the frame of the camera. Connect them to DigitalPin 11 and to ground.
Fire away little guy!

Optional gimmick.
We can put a red LED under the coinslot. This will be a fun little gimmick for when it is dark.
Connect a wire to DigitalPin 8 and put a 220Ω resistor between the LED and the wire to prevent it from blowing up. Connect the short pin of the LED to ground.

Step 4: The C++ Code

Step 4.1: Setting up the main.cpp code
4.1.1: Download "main.cpp" and copy the code to your own main.cpp.
4.1.2: On line 14 change the "com" to the com the arduino uses. "\\\\.\\COM(change this)"
4.1.3: On line 21 and 22 set the right path tho the files "haarcascade_frontalface_alt.xml" and "haarcascade_eye_tree_eyeglasses.xml"
If openCV is installed on the C drive, these files can be located here: "C:\\opencv\\build\\etc\\haarcascades\\" Keep the double backslashes or add one where there only is one.

Step 4.2: Add tserial.h and Tserial.cpp
These 2 files will take care of the communication between the arduino and the PC.
4.2.1: Download tserial.h and Tserial.cpp.
4.2.2: Place these 2 files in the project directory. In the Solution explorer right-click on the project and select add > existing item. In the popup window select the two files to be added.

Step 4.2: Add CXBOXController.h and CXBOXController.h
These files will take on the controller part of the project.
4.2.1: In the Solution explorer right-click on the project and select add > existing item. In the popup window select the two files to be added.
The C++ files are setup.

Step 5: The Arduino Code

Step 5.1: NewPing library
5.1.1: Download the ArduinoCode.ino and open it in the arduino IDE.
5.1.2: Go to "Sketch > Include libary > Manage libaries" .
5.1.3: Search in the filter box to "NewPing" and install this library.

Step 5.2: Pitches library
5.2.1: Download pitches.txt and copy the content of pitches.txt.
5.2.2: In the Arduino IDE press CTRL+Shift+N to open a new tab.
5.2.3: Paste the code from the pitches.txt in to the new tab and save it as "pitches.h".
The Arduino code hase been set up.

Step 6: 3D Printing and Refining the Print

Step 6.1: Print the 3D file
Open the printfile.form and check if everything is okey. If everything seems okey, send the printjob to the printer.
If something seems of or you want to change the model. I have included the 3Ds Max files and OBJ files for you to edit.

Step 6.2: Refine the model
6.2.1: After printing is done soak the 2 models in some 70% alcohol to remove any print resedu.
6.2.2: After printing lay the model in the sun for a few hours to let the UV-light harden the model. Or you can use a UV-lamp to harden the model. This needs to be done because the model will be sticky.

6.2.3: Remove the support framework. This can be done with a wire cutter. Or any other tool that can cut plastics.
6.2.4: Some parts of the 3D print can still be soft. Even if the model has been in a lot of UV-light.
The parts that can be soft, are the parts that where close to the support frames. Lay the model in more sun of UV-light to harden out.
6.2.5: With a "dremel" you can sand away all the little bumps made by the framework. You can try to fit the servos in to the frame. If they won't fit you can use the Dremel to sand material away. make it fit.

Step 7: Building the Box

Step 7.1: Making the holes
I have included a blueprint of the box in question. The blueprint is not up to scale, but all the sizes are correct.
7.1.1: Start by marking all the holes on the right locations.
7.1.2: Drill all the holes. The bigger holes can be made up to size with a Dremel.
7.1.3: The square holes can be drilled as well. But to make them square you can fit the Dremel with a small file and file out sharp corners.
7.1.4: Try to fit all the components. if they fit, you are good to go!
7.1.5: Watch out for wood splinters. Use sand paper to get rid of them.

Step 7.2: Painting
7.2.1: Start with sanding the lid. We need the paint to stick.
7.2.2: Take a cloth and put a little bit of Turpentine on it to clean the box.
7.2.3: Now you can spray paint the box whatever color you desire.

Step 8: Finishing Up.

Now we have to put everything in to place and let it do it's thing.
Step 8.1: The coin detector
8.1.1: Glue down some metal braces for the coin detector.
8.1.2: Solder each wire from the connector to a brace.
8.1.3: Test the connection with a coin. If there is no closed circuit, solder the wires more to the edge.

Step 8.2: The protoboard and RGB leds
8.2.1: Place the protoboard in the top right corner and tape it down!
8.2.2: Connect the RGB leds with the wires from the protoboard!
8.2.3: Connect all the wires from the protoboard tot the arduino.

Step 8.3: The sonar sensor
8.3.1: Place the sensor in the holes we made for it.
If you do have male to female jumpwires you can skip 8.3.2
8.3.2: Cut some male and female wires in half and solder the female and male wires together to make a single cable we can use to connect the sensor to the arduino.
8.3.3: Connect the sensor to the arduino

Step 8.4: Lasers and camera
8.4.1: Glue the small frame to the camera. Make sure it is upright.
8.4.2: Put the lasers in to the frame as well. Glue them down so the enemy won't steal them!

Step 8.5: The Servos and 3D print
8.5.1: Glue the servo in the hole of the lid
8.5.2: Upload the arduino file to the arduino (this makes the servos stand in the right position)
8.5.3: With the servo came a little round plateau. Place this on the servo in the lid.
8.5.4: Put the large 3D print on the servo and plateau and screw them tightly together with a screw.
8.5.5: Place the second servo on the small 3D print and glue them together.
8.5.6: Put the camera in place and every thing is ready to go!

Step 9: Start the Program

To start the robot open the C++ File in Visual studio. Make sure you are in "debug mode"
Upload the arduino file to the arduino. Once that is uploaded press play in visual studio.
And the robot will fire and collect all the coins in the world!!!