Introduction: WebRTC Creeper Drone - Browser Controlled RC Car

This is an advanced Instructable, which describes the process of building a drone controlled remotely from a browser. The drone uses an Android phone to stream video and audio back to the browser via WebRTC.

Since the software used in this project is somewhat complex, you will need to have some programming experience in order to benefit from this Instructable, as I will not be explaining the software parts in great detail.

Step 1: ​Allow Me to Introduce the Creeper Drone

Creeper Drone was created from a cheap RC truck, which I modified with an Android and Raspberry Pi, so it can now be driven over a WiFi network from any browser that supports WebRTC. The Creeper transmits a video stream, allowing the driver to control the Creeper from a remote location. Bi-directional audio is also supported, providing the driver with the ability to converse through the Creeper.

This Instructable provides a detailed description of the hardware, software and part manufacturing that went into this project.

Step 2: Architecture and Design

All 3 devices must be connected to the same WiFi network. The diagram above shows the Android phone being used as a WiFi Hotspot, with the other 2 devices connecting to it, but other configurations are possible.

One obvious improvement to this design would be to enable communication with the Android device via the Mobile connection instead of WiFi, thus extending the drone's range to anywhere with LTE coverage. This would likely require an addition of a TURN server into the design.

The Creeper Drone can be controlled from any WiFi connected device that can run Chrome, or another WebRTC capable browser.

A WebSocket is used as a signalling channel between the browser and the Android phone.
A simple TCP/IP socket is used for communicating with the Raspberry Pi. The Raspberry Pi uses the onboard GPIO port to communicate with the Control Board which handles the hardware requirements of driving servos and motors.

Communication with the drone begins by opening the browser and connecting to http://[android-ip-address]:8000/. This causes the embedded Web Server (Netty) running inside the Android App to serve an HTML file and some JavaScript code necessary for handling the browser side functionality. This includes processing of keyboard events to send Drone Control commands, display of Creeper Status, WebRTC signalling through the WebSocket, display of the Media Stream from the drone and handling of the Media Stream to the drone.

Once the WebSocket handshake completes, the WebRTC stack on the Android is initialized and an SDP Offer is sent via the WebSocket to the browser. The client code running in the browser receives the Offer, initializes its own WebRTC stack and send an SDP Answer. At this point the Android and the Browser exchange ICE Candidates and establish a WebRTC session.

When a successful WebRTC session is established, the user can now see and hear everything through the Creeper Drone. Using the keyboard the user can now control the camera position and drive the drone.

A Control Command is sent through the WebSocket, recognized as such by the CreeperAndroid app, and forwarded to the Raspberry Pi via a network socket. The RPi receives the command and dispatches it to the correct PWM Generator module (servos require a different control scheme from DC motors). The selected PWM module uses the pigpio library to send PWM signals through the GPIO port. Once the selected command was executed, a Status Message for the appropriate servo/motor is communicated back through the network socket, through the WebSocket and into the Browser page the user is using to control the drone.

Step 3: Selecting the RC Car

The RC car I used was a $50 New Bright F-150 Truck.

I highly recommend not getting the same RC truck! I only got it b/c at the time, I did not know any better. The problem with this toy is illustrated in the close up of the steering mechanism. That small gear shaft is not driven by a servo - it's a conventional motor, which spins until it is jammed at the extremes of travel. The wheels are straightened when power is removed by a small spring on the other side. This means that there is only one angle at which the wheels can be turned, and that angle is way too small for making sharp turns at low speeds.

Even worse, this design assumes that the car will be instructed to turn only momentarily, b/c it does not have a speed controller and therefore goes way too fast to ever require long turns. Since we are definitely building a speed controller into the Creeper Drone, the time we need to spend turning increases drastically. Since the turning mechanism relies on stalling a DC motor (which should never be done anyways), we spend way too much time drawing about 2A from the battery, which naturally destroyed the steering motor in about 5 minutes of heavy usage.

I actually had to completely remove the steering mechanism and replace it with a servo mechanism that I will discuss later.

If you would like to try this project, do yourself a favour and select some kind of treaded platform that can turn on a dime, such as an RC tank of comparable size.

Step 4: Major Parts List

The major components you will need:
  1. A large RC car or tank
  2. A battery pack to power the Raspberry Pi and the Servos. I used a RAVPower 10400mAh External Battery Pack.
  3. A Raspberry Pi.
  4. An Android device with some sort of holster and clip that will help you attach the Android to the servo assembly. I used a Samsung Galaxy S4 running Android 4.2.2, with an Otterbox Defender.
  5. Servos powerful enough to hold the weight of the phone and handle the front steering. I used the HS-645MG Ultra Torque Servo. This servo is strong enough, but it still fails to park properly in positions where the weight of the phone is torquing on it with full force.
  6. Pan and Tilt unit. I got the SPT100.

Step 5: Assorted Parts List

Some other things you will need:

  1. Materials for making a PCB. There are plenty of tutorials out there, so I won't cover it here.
  2. Type A Male to Type A Male USB cable for powering the servos.
  3. Type A Male to Micro USB Male for powering the Raspberry Pi.
  4. L293D Half-H Driver (x2)
  5. GPIO connection header, socket and 13x2 ribbon cable.
  6. 680uF capacitors (x3)
  7. Assorted resistors, caps and transistors shown in the schematic later on.

Step 6: Install Python and Pigpio Library on Raspberry Pi

TODO: Python install procedure

TODO: pigpio install procedure

You may be wondering why I chose pigpio, instead of the more well known RPi.GPIO library. The problem with RPi.GPIO is that it still does not support hardware PWM on more than one pin, whereas pigpio provides independent hardware PWM on any of GPIO pins simultaneously by utilizing the DMA controller on the RPi.

Step 7: Control Board

This section describes the control circuit used to drive servos and the rear motor from RPi's GPIO port.

Although it's way to early in the process to be making a board, I am placing this section here b/c I wanted to show you the schematic. It will be useful to have parts of this circuit wired up on a breadboard in order to incrementally test the different parts of the software that follow.

The board was designed in Kicad (professional quality, free, open-source software) and made on ProtoMat S103 available at my local Maker Space (AssentWorks). I am attaching my Kicad project to this tutorial.

Note that the connector for the rear motor was taken from the original board that was inside the truck.

The shape of the board was made to exactly reproduce the shape of the original board. This way I was able to reuse the existing mounting posts and attach to the 9V battery posts.

The circuit is fairly self-explanatory, I hope. The NPN transistor hooked up to the Rear_Direction output is acting as a simple inverter.

If you have any questions about the schematic or the board, feel free to ask.

Step 8: Taking Apart the Car and Reverse Engineering Motor Control

The F-150 was very easy to take apart. Everything is held with screws and all important parts are removable.

Make sure you keep the original board after you take it out, since it can be very useful for testing when you are debugging or reverse engineering the controls. You can also take the connectors from the original board and put them onto yours.

Turns out that the rear motor could be successfully throttled with a simple PWM technique. I had to write a small PWM generator utility which was very useful for figuring out how to control the motor and the servos. You can download the utility from here:

https://github.com/vace117/CreeperPiController/tre...

Make sure that you grab the whole directory though, since the pulse_generator.py script has dependencies.

Watch the video and read the code to learn how to use the script.

You will need to install Python and pigpio on your Raspberry Pi before you can do this.

Step 9: Get Your Servos Working

Now that you have the rear motor working, time to play with the servos.

You can use the following test script https://github.com/vace117/CreeperPiController/blo..., just make sure that your servos are hooked up to the correct GPIO pins. Look at the source code to figure out which.

The video shows me activating the servos from my phone, b/c I am SSHed into the RPi from it.

Step 10: WebRTC

WebRTC is a very cool technology for real-time peer-to-peer media exchange. If you are not familiar with it, here are some useful links:

http://www.html5rocks.com/en/tutorials/webrtc/basi...

https://www.webrtc-experiment.com/docs/webrtc-for-...

https://www.webrtc-experiment.com/docs/WebRTC-Peer...

http://www.webrtc.org/reference/architecture

Very useful read: http://dev.w3.org/2011/webrtc/editor/webrtc.html

Also, check out Google's demo: https://apprtc.appspot.com of this technology in action.

Step 11: Compiling WebRTC for Android

You can either try using the libraries I compiled for my project, or you can try to build your own. If you want to reuse my binaries, be aware that I was using WebRTC r5248, which is pretty old by now. Also, since WebRTC is not yet stable, you will not be able to use my binaries with the latest version of Chrome! I tried that and it does not work, due to some incompatible changes to the WebRTC in the browser. I can confirm that my binary works on Chrome Version 31.0.1650.63.

You can download my ADT project with the libraries, all ready to go onto your phone, from here: https://github.com/vace117/CreeperAndroid

If you want the latest version, you'll have to get the WebRTC code and compile it for your Android. The instructions for doing so are found here:

http://www.webrtc.org/reference/getting-started

Everything you need to know is there, but this is still probably the most confusing step. I am attaching a small script I wrote that sets up the environment for a proper Android compile.

Other than that, you'll just have to familiarize yourself with Google's build tools a bit.

Once your build succeeds, you will end up with the following two files that you need for your Android app:

  • out/Debug/libjingle_peerconnection.jar
  • out/Debug/libjingle_peerconnection_so.so

These will go into the libs/ folder of your Android app.

Please note that libjingleMUST be compiled on a 64-bit OS.

Step 12: Troubleshooting WebRTC on Android

Getting WebRTC to work was not very straightforward, so here are a few notes to help you.

First, I am attaching SDP Offer and Answer from a working system. If something doesn't work for you, comparing these to yours might provide some insight. Note that there is some magic going on in the answer. It turned out that the Opus audio codec which WebRTC tried to use by default, does not work on Android. Possibly b/c Opus requires raw audio sampled at 48kHz, whereas the Android provides 16kHz. Trying to use Opus results is a SEGFAULT somewhere in libjingle. In order to avoid the problem entirely, I decided to modify the Answer to force the use of ISAC/16000 instead of Opus. This was achieved with some Javascript code you can look at here.

Next, you need to know about some misleading errors that you will see in the libjingle LogCat output. You can safely ignore these:

E/libjingle( 9365): Error(webrtcvideoengine.cc:1410): webrtc: (voe_audio_processing_impl.cc:1001): virtual int webrtc::VoEAudioProcessingImpl::SetTypingDetectionStatus(bool): not supported

E/libjingle( 9365): Error(webrtcvideoengine.cc:1410): webrtc: (voe_hardware_impl.cc:443): virtual int webrtc::VoEHardwareImpl::SetPlayoutDevice(int): not supported

W/libjingle( 9774): Warning(channelmanager.cc:252): Failed to SetAudioOptions with microphone: speaker: options: AudioOptions {aec: true, agc: true, ns: true, hf: true, swap: false, typing: true, conference: false, agc_delta: 0, experimental_agc: false, experimental_aec: false, aec_dump: false, experimental_acm: false, } delay: 0

E/libjingle( 9365): Error(webrtcvideoengine.cc:1410): webrtc: (voe_volume_control_impl.cc:545): virtual int webrtc::VoEVolumeControlImpl::SetOutputVolumePan(int, float, float): not supported

Step 13: Android App (CreeperAndroid)

Load this app onto your phone: https://github.com/vace117/CreeperAndroid

Once the app starts, it will tell you what IP addresses it is listening on. Hit any of those from a Chrome browser, and you should see the video stream from the phone on that page.

Be careful! If your phone is too close to your computer speakers, you are going to get a seriously unpleasant feedback loop going. You might want to lower the volume or mute the sound entirely, until the Creeper Drone is safely deployed in the field away from the Command Center.

If everything went well, the CreeperAndroid app screen will switch to a green Creeper picture. You can still access the detailed log by swiping to the right on the screen.

If you don't get a connection, hook up the phone to your development workstation via USB, start ADT and launch the app while monitoring the LogCat view. Also, open the console in Chrome - there is a lot of useful logging there too.

Step 14: Python App (AndroidPiController)

Do not do this step unless you already have at least one of the following scripts working properly:

If these work and control your hardware as expected, proceed with the installation of CreeperPiController:

https://github.com/vace117/CreeperPiController

  1. Git clone the project
  2. Open AndroidCommandSocket.py and set ANDROID_HOST so it properly points to your CreeperAndroid on the phone
  3. Start the CreeperAndroid app on your phone
  4. Start "./AndroidCommandSocket.py"

If all goes well, you should see your servos move to the default position, and there will be message in the browser log saying: "CREEPER_READY"

Now you can control your motor and servos through the browser!

Step 15: Control Scheme

The controls are as follows:

  • "LOOK_LEFT", // 'a'
  • "LOOK_RIGHT", // 'd'
  • "LOOK_DOWN", // 'w'
  • "LOOK_UP", // 's'
  • "LOOK_CENTER", // 'c'
  • "ACCELERATE", // UP Arrow
  • "REVERSE_ACCELERATE", // DOWN Arrow
  • "STOP", // Space
  • "WHEELS_LEFT", // LEFT Arrow
  • "WHEELS_RIGHT", // RIGHT Arrow

Step 16: Replacing the Steering Mechanism

The steering mechanism had to be completely replaced for reasons described in "Selecting the RC car" step.

My solution is nothing more than a quick hack, but it seems to work pretty well.

  1. Open the gearbox as shown in the picture and remove all the gears and the DC motor.
  2. Reassemble the gear box w/o any gears, but keep the red pin and the centering spring in place!
  3. Close the gearbox.
  4. Hot-glue a servo on top of the gear box and attach the wires to the wheels, as shown in the picture.

Done! We now have a MUCH more efficient steering mechanism than the original.

Step 17: Printing the Servo Assembly Receiver

Now we have to attach our two-servo assembly to the front of the Creeper Drone. There are many different ways to solve this problem, but I chose to treat it as an opportunity to learn about 3D printing, since AssentWorks offers a few 3D printers to their members.

I used SolidWorks to model the front of the car, and then designed a receiver for the servos that fits perfectly around the existing car shape.

Bonus: Turns out printed plastic is really good at gripping screws - those screws are solid in there!

I am attaching the 3D model to this step.

The printer I used is Stratasys uPrint SE.

Step 18: Printing the "Trunk"

The Control Board attaches naturally, b/c I made it the same shape and size as the original. But what about the battery pack and the RPi? For this I printed another 3D part.

This part was printed on Stratasys Fortus 400mc.

3D Model attached.

Step 19: Creeper Startup and Shutdown Procedure

There is a fairly serious problem with this design that I did not account for.

When the Raspberry Pi is either booting or shutting down, the GPIO pins tend to produce quite unexpected signals. With the current design this happens to turn on the rear motor to full throttle and keep it there :). So if you are not careful, your Creeper Drone is going to run away from you and smash into something really hard.

To prevent this from happening please follow this shut-down procedure:

  1. Unplug the GPIO ribbon cable.
  2. Unplug the USB cable that powers the servos.
  3. Issue the 'poweroff' command to the RPi and wait for it to shut down gracefully.
  4. Unplug the USB cable that powers the RPi.
  5. Take out the 9V battery.

To start the Creeper:

  1. Plug in the 9V battery. The RED led on the side should come on.
  2. Make sure that you WiFI network is available. If you are using the Android hot-spot, start that now.
  3. Plug in the RPi power USB cable.
  4. Plug in the servos power USB cable.
  5. Start CreeperAndroid app on the phone.
  6. Mute the sound on Command Center computer or device.
  7. Connect the Command Center browser to the phone and make sure that video stream is working.
  8. Plug in the GPIO ribbon cable into the RPi.
  9. SSH into the RPi and start ./AndroidPiController. This should move the servos into "center" position.
  10. Attach the phone to the Creeper Drone.
  11. Deploy the Creeper Drone and begin away mission :)

That's it. Happy Creeping!