Virtual Presence Robot

This mobile robot interacts with its physical surrounding, by representing the "virtual presence" of the person controlling it remotely. It can be accessed by anyone, anywhere in the world, to dispense treats and play with you.

The work here is developed by two people (one in Germany and one in the US) as an attempt to move beyond traditional means of internet-based communication by creating a physical interface for remote interaction. As COVID-19 continues to affect the world, and everyone is responsible for limiting our physical exposure to people, we try to bring back the tangible connection that is a part of physical interaction.

It is based on the ESP32-Camera-Robot-FPV-Teacher-Entry Instructable and modified to include distance sensor, treat dispenser and "control from anywhere in the world" capability, provided you have a somewhat stable internet connection.

Supplies

The project has 4 main parts - a Mobile Robot Car, a Chip Dispenser, a Joystick, and Network Communication Setup.

Mobile Robot Car

• Breadboard
• 2 Wheel Drive Motor and Chassis Robot Kit (includes wheels, DC motors, mounting board and screws)
• Arduino Mega 2560 (if you build without the distance sensor or the Chip Dispenser an Uno will have enough pins)
• (3) 9V Batteries (have a few more around as you will drain them debugging)
• LM2596 Power Supply Module DC/DC Buck 3A Regulator (or similar)
• ESP32-CAM Wifi Module
• FT232RL FTDI USB to TTL Serial Converter (for programming the ESP32-CAM)
• HC-SR04 Ultrasonic Distance Sensor
• L298N Motor Driver
• (3) LEDS (any color)
• (3) 220 Ohm Resistors

Chip Dispenser

• (2) SG90 Servos
• Cardboard / Paperboard

Joystick

• Arduino Uno
• Joystick Module
• Mini Breadboard, (1) LED, (1) 220 Ohm Resistor (optional)

Other

Lots of Breadboard Jumper Wires
Extra Cardboard / Paperboard
Tape
Scissors
Ruler / Measuring Tape
Small Philips Screwdriver
Small Flathead Screwdriver

Patience =)

Step 1: Mobile Robot Car

The Robot Car chassis serves as a mobile platform, with an Arduino MEGA as the main micro-controller driving the motors, reading sensor values and actuating the servos. Most actions are performed by having the Arduino MEGA receive commands via serial communication, sent from the ESP32-CAM. While ESP32 provides camera live stream to control the robot, its other function is to manage a wireless connection between the robot and the server, therefore allowing users to control it from anywhere in the world. The ESP32 receives commands from the web page via keypress and sends them to the Arduino MEGA as char values. Based on the value received the car will go forwards, backwards etc. Because remote control via internet is dependent on a lot of external factors including high latency, poor stream quality, and even disconnections, a distance sensor is incorporated to keep the robot from crashing into objects.

*Due to the high and fluctuating power requirements of the ESP32 chip a Power Supply Regulator is recommended for use with battery power (see wiring diagram).

Step 2: Mobile Robot Car - Circuit Diagram

We will walk you through assembling this step by step.

Step 3: Mobile Robot Car - Assembly (Motors)

After you assemble the 2WD chassis, we start by connecting the motors and battery to the Arduino MEGA through the L298N driver.

Step 4: Mobile Robot Car - Assembly (Distance Sensor)

As there are quite a few components to connect, let's add a breadboard, so we can connect the power and shared ground more easily. After we re-organize the wires, connect the distance sensor and fix it in the front of the robot.

Step 5: Mobile Robot Car - Assembly (ESP32 CAM)

Next, connect the ESP32-CAM module, and fix it next to the distance sensor near the front of the robot. Remember this rather power-hungry component requires its own battery and a DC-regulator.

Step 6: Mobile Robot Car - Assembly (Chip Dispenser)

Now, let's add the chip-dispenser (more on this in the "Chip Dispenser" section). Wire up the two servos according to the Fritzing diagram, and fix the dispenser at the tail of the robot.

Step 7: Mobile Robot Car - Assembly (Cookies!)

Finally, we add treats to the dispenser!

Step 8: Mobile Robot Car - Arduino Code

RobotCar_Code is the code you will need to load on the Arduino Mega.

Here is how it works: the Arduino listens to bytes that are sent from the ESP32 via serial communication on the 115200 band. Based on the byte received, the car will move forward, backward, left, right etc. by sending either a HIGH or LOW voltage to the motors to control direction, as well as a PWM variable between 0-255 to control the speed. To avoid collisions this code also reads the values coming in from the distance sensor and if the distance is less than a specified threshold, the robot will not move forward. Lastly, if the Arduino receives a command to dispense a treat, it will activate the servos in the Chip Dispenser.

Step 9: Mobile Robot Car - ESP32 Code

The ESP32 allows for communication between the server and Arduino via Wifi. It is programmed separately from the Arduino, and has its own code:

• ESP32_Code.ino is the code for ESP32 to send information to the Arduino
• app_httpd.cpp is the code needed for the default ESP32 webserver and set the function to listen for keypresses. Good for debugging and testing on local wifi. It is not used for communication outside the local network.
• camera_index.h is the html code for the default web application
• camera_pins.h defines the pins depending on the ESP32 model

The ESP32 code uses the Wifi library as well as the ESP32 Add-on, which can be installed in the Arduino IDE by following these steps:

1. In the Arduino IDE go to File > Preferences
2. Then in the Settings tab under Additional Boards Manager URL enter the following "https://dl.espressif.com/dl/package_esp32_index.json"
3. Now open the Boards Manager and go to Tools > Board > Boards Manager and search for the ESP32 by typing "ESP32"
4. You should see "esp32 by Espressif Systems". Click Install.
5. Now the ESP32 add-on should be installed. To check go back to the Arduino IDE and go to Tools > Board and select the "ESP32 Wrover Module".
6. Again go to Tools > Upload Speed and set it to "115200".
7. Lastly, go to Tools > Partition Scheme and set it to "Huge APP (3MB No OTA/1MB SPIFFS)
8. Once you have completed this, I recommend following this tutorial by RandomNerdTutorials which explains in detail how to finalize setting up the ESP32 and upload code with the FTDI Programmer
   
   Programming the ESP32

Step 10: Chip Dispenser

The Chip Dispenser is an inexpensive addition to the mobile robot allowing it to affect the local environment and interact with people / animals by leaving a tasty treat. It consists of a paperboard exterior box with 2 servos mounted inside, as well as an interior paperboard cartridge which holds items (such as candy or dog treats) to dispense. One servo acts as a gate while the other pushes the item out.

*All dimensions are in millimetres

Step 11: Joystick

While it can be fun to control a robot with the keyboard, it's even more fun and intuitive to use a joystick, where the robot reacts directly based on the direction you push. Since this robot is actuated via keypresses recorded on the webpage we needed our joystick to emulate a keyboard. This way users without a joystick can still control the robot directly from a keyboard, but others can use the joystick.

For this we only had an Arduino Uno which does not have the ability to use the library so we programmed it directly using the a USB protocol known as Device Firmware Update (DFU) which allows the arduino to be flashed with a generic USB HID keyboard firmware. In other words, when the arduino is plugged into the usb it is no longer recognized as an arduino but as a keyboard!

Step 12: Joystick - Circuit Diagram

Here is how we wired up the joystick.

Step 13: Joystick - Keyboard Emulator

In order for your Arduino Uno to emulate a keyboard, you need to directly program the Atmega16u2 chip on the Arduino via a Manual Device Firmware Update (DFU). The following steps will describe the process for a Windows machine, and hopefully help you avoid some of the issues we ran into.

The first step is to manual write the Atmel usb driver to the Arduino so it is recognized as a USB and not an Arduino which enables it to be flashed with the FLIP programmer.

1. Download Atmel's FLIP Programmer from here
2. Plug in your Arduino Uno
3. Go to Device Manager and find the Arduino. It will be under COM or Unknown Device. Plug it in and out to ensure this is the correct device.
4. Once you have found the Arduino Uno in the Device Manager, right click it and select properties > Driver > Update Driver > Browse My Computer For Driver Software > Let me pick from a list of available drivers on my computer > Have Disk > Browse to the file "atmel_usb_dfu.inf" and select it. This should be in the folder where your Atmel FLIP Programmer was installed. On my computer it is here: C:\Program Files (x86)\Atmel\Flip 3.4.7\usb\atmel_usb_dfu.inf
5. Install the driver
6. Now go back to the Device Manager you should see an "Atmel USB Devices" with the Arduino Uno now labeled as an ATmega16u2!

Now that the computer recognizes the Arduino Uno as a USB Device we can use the FLIP Programmer to flash it with 3 separate files and turn it into a keyboard.

If you unplugged your Arduino Uno after the first part, plug it back in.

1. Open FLIP
2. Reset the Arduino Uno by briefly connecting power to ground.
3. Click Device Selection (icon like a microchip) and select ATmega16U2
4. Click Select a Communication Medium (icon like a usb cord) and select USB. If you completed the first part correctly the other greyed buttons should become usable.
5. Go to File > Load Hex File > and upload the file Arduino-usbserial-uno.hex
6. In the FLIP window you should see three sections: Operations Flow, FLASH Buffer Information, and ATmega16U2. In the Operations Flow check the boxes for Erase, Program, and Verify, then click Run.
7. Once this process completes, click Start Application in the ATmega16U2 section.
8. Plug cycle the arduino by unplugging it from the computer and plugging it back in.
9. Reset the Arduino Uno by briefly connecting power to ground.
10. Open the Arduino IDE and upload the file JoyStickControl_Code.ino to the board.
11. Plug cycle the arduino by unplugging it from the computer and plugging it back in.
12. Reset the arduino by briefly connecting power to ground.
13. Go back to FLIP, make sure Device Selection says Atmega16U2
14. Click Select a Communication Medium and select USB.
15. Go to File > Load Hex File > and upload the file Arduino-keyboard-0.3.hex
16. In the FLIP window you should see three sections: Operations Flow, FLASH Buffer Information, and ATmega16U2. In the Operations Flow check the boxes for Erase, Program, and Verify, then click Run.
17. Once this process completes, click Start Application in the ATmega16U2 section.
18. Plug cycle the arduino by unplugging it from the computer and plugging it back in.
19. Now when you go to Device Manager there should be a new HID Keyboard Device under Keyboards.
20. Open a notepad or any text editor and start moving the joystick. You should see numbers being typed!

If you want to change the code in the Arduino sketch, for example writing new commands to the joystick, you will need to flash it with all 3 files each time.

Some useful links:
Arduino DFU
AtLibUsbDfu.dll not found

This keyboard emulator is based on this tutorial by Michael on June 24, 2012.

Step 14: Network Communication

To receive video stream and send commands to the robot from anywhere in the world, we need a way of getting data to and from the ESP32-CAM. This is done in two parts, a connection handler on your local network, and a public server. Download the three files to achieve this:

• Handlers.py: relays information from ESP32-CAM and the public server (tested on Python 3.8)
• Flask_app.py: defines how your app responds to incoming requests.
• Robot_stream.html: renders video in your browser and listens for commands through keyboard / joystick (tested on Chrome)

Connection Handler
You can code this directly in app_httpd.cpp, but for easier debugging we use a Python script running on a PC connected to the same network. Open handlers.py and update the IP address and username to your own, and you’re ready to go. The stream will start when you run this file.

Public Server
To access everything on the internet, you can start a server with a PaaS of your choice. On pythonanywhere (PA) setting this up takes less than 5 minutes:

1. Sign up for an account and log in
2. Go to the “Web” tab and hit “Add a new Web App”, choose Flask and Python 3.6
3. Copy flask_app.py into /mysite directory
4. Copy robot_stream.html into /mysite/templates directory
5. Click “Reload”

And … you’re all set!

Disclaimer:
This networking workflow is quick and simple but very far from ideal. RTMP or sockets would be more appropriate for streaming, but they are not supported on PA and requires some experience with networking and server setup. It is also recommended that you add some security mechanism to control access.

Step 15: Putting It All Together

Now, turn on your robot, run handlers.py on a computer (connected to the same network as the robot), and you can control the robot from a browser based on the url you set from anywhere you want. (e.g. https://your_name.pythonanywhere.com/robot_name)