First-person WiFi Driving of Roombas and Other Infrared RC Vehicles

Sticking a smartphone a remote controlled vehicle and having it stream camera video over WiFi lets you have lots of fun driving the vehicle with its remote while viewing images from it on another device and using the vehicle's own remote control. You can even do this where there is no WiFi access point with a hotspot on the phone or laptop (even if the phone or laptop has no Internet connection). And you can control your phone's flash for use as a headlight. All this uses simple off-the-shelf software, but I'll describe it in the first two steps.

Problem: But what if your vehicle is, like the Roomba, controlled by a line-of-sight (or close to line-of-sight: the light will bounce off walls to some degree) infrared remote? Sure, you could make sure that your vehicle is always within sight, but then what's the point of first person driving if you can see the vehicle?

Solution: Stick an infrared emitter dongle in the phone's audio jack and make an open source app -- IR Server -- that lets you control the vehicle (or any other IR-controllable device) via a web browser. Now you can use your laptop (or tablet or other device) both to steer and to view the video.

That's what I did, using an Android phone with two infrared controlled vehicles: a Roomba 530 and a bulldozer from the kids' Thames and Kosmos Remote-Control Machines Set. And of course now that I've done it, you don't need to write the app. If you just want to drive a 500-series Roomba or a Thames and Kosmos machine, you can just use my IR Server app. If you want to drive some other infrared-controlled vehicle -- or just control your TV via a browser! -- then you just need to modify one of the html files that IR Server uses for controlling.

My IR Server app is still a work in progress, which is why I am only making an apk available on github rather than putting it on Google Play as yet. I have the occasional hiccup with it.

What you need:

• Android device with camera
• Webcam app: I use the free version of IP Webcam; if you prefer open source, there is SpyDroid
• IR Server app (source code and binaries here)
• Audio jack infrared emitter: I used one from a defunct Griffin HeloTC; you can buy a cheap IR blaster; you can also make your own but the Griffin worked more reliably for me (newer phones have built-in support, but I don't have such a phone so I didn't add support for that)
• Remote controlled toy
• Optional: Way of capturing IR signals (I won't describe this, as the hardware you have probably differs a lot from mine)

Step 1: First-person View Only: Simple Version

The simplest first-person driving solution when you have a WiFi connection is just to attach a phone running webcam server software (I use IP Webcam; the free version does the job) to the toy. My Galaxy S2 phone fits neatly behind the bumper of my daughter's toy truck, and a single rubber band holds it nicely in place.

To view the images from the webcam for first-person driving, configure your webcam server software. I recommend using no higher than 640x480 resolution, a fairly low quality (30-50 in IP Webcam), and a frame rate of no more than about 20. With higher settings, lag can be annoying. Make sure the port number is set to 8080 (this will matter for IR control). Then start the webcam server (big button at the bottom of the settings screen in IP Webcam) and it should show on the screen the address to put into the other device. Then just point a web browser on a computer, tablet or phone on the same WiFi network to that address (with IP Webcam, you'll have to choose "Browser" or "Flash" or some other option--I recommend "Flash"--before you see the video). Or you can use a webcam viewing app like tinyCam Monitor (the free version works fine for me) on an Android device.

Then use the original remote control for your toy along with the device you're viewing the driving in.

A cool thing about IP Webcam is that it supports turning on your phone's flash, which you can use as a headlight for exploring dark closets. (In tinyCam Monitor you can activate this option by touching the horizontal three dots until it appears. In a web viewer, there is a button for that.)

Step 2: First-person View Only: Where There Is No WiFi

It's a bit more challenging if you're away from a WiFi network. I've managed to get it working in two different ways, and there is a third I haven't tested yet. I'm not going to give the tap-by-tap details, because it's going to be different on different devices and different OS versions, but I'll be happy to share more details as needed.

Method 1: Run a WiFi hotspot app on the webcam phone. Then connect the other device's WiFi to the hotspot and go to the address the webcam app shows. You don't actually need Internet service on the phone that generates the hotspot. The phone I was using as a webcam was my old Android 2.3 Galaxy S2 without the phone service any longer operating. I simply launched the Sprint Hotspot app and enabled the hotspot in the absence of any Internet connection. Normally, Sprint charges for hotspot usage, but since the phone isn't connected to the Sprint network any more, the point seemed to be moot. My newer phone then happily connected to that hotspot and I was able to view the camera stream without any problems using the IP address that the webcam software showed on screen. To avoid tethering charges, if you're using a phone with active service for the webcam, I recommend turning off mobile data. This leads to a possible hang-up: one hotspot app I tested (FoxFi) refuses to start the hotspot in the absence of an Internet connection. You may be able to fool it, though: just start FoxFi when you've got Internet, e.g., on your home WiFi network, and then keep it operating.

Method 2: Take your laptop into the field, turn it into a virtual router and connect all your devices to it. You don't need Internet there. You can then either view the video stream on the laptop or on some other connected device. In theory, you should be able to do this in an easy and friendly way with the Virtual Router program, but that didn't work for me on my Windows 8 laptop. What worked much better were the instructions here (since you are doing it for use in the field where you don't have Internet, you can skip the first part of the instructions about sharing the connection, and just start where it says "Now right-click in the bottom left of your screen"; to terminate the connection, run netsh wlan start hostednetwork in your administrator shell); for Windows 7, see instructions here. No doubt similar things work on Linux and OS X.

Step 3: Infrared Over WiFi: Background

The simple WiFi webcam and viewer solution is great for driving radio-frequency controlled toys. But things get more complicated if you want to drive something that uses an infrared remote, and you want to be able to drive around corners.

The trick I used was to connect an infrared emitter to the phone that was running the webcam server software and to make an infrared server app (IR Server) that lets one control the emitter via a web interface. The idea is that the phone that serves as the camera also has the infrared emitter, and runs both webcam software and my infrared server app. Then you can connect via a browser on another device (I used a laptop) to IR Server, and click through to a page that includes both controls for the infrared toy and the image through the webcam.

Feel free to skip the following technical details.

The infrared emitter fits into the phone's stereo jack. The way this works is that the phone encodes the infrared control signals as audio, with the carrier frequency halved but everything else in the signal unchanged. The infrared emitter then doubles the frequency (e.g., by using two LEDs), and perhaps adds amplification.

To make my web-based infrared control app, I started with the code for the simplest open source Android web server I could find, and modified it to play audio-encoded infrared signals when a client's request includes an ?ir=xyz query where xyz is a description of the infrared signal to be sent (I will explain these in a later step). Full source code is here.

I also made a few other changes to the web server app (the server now runs as a Service instead of as an Activity and it supports persistent connections which are crucial for this real-time application), and included in it some simple pre-built web pages for controlling Roomba 500-series vacuums and Thames and Kosmos remote-controlled toys, as well as for viewing them alongside the output of a webcam server.

IR Server has built-in support for Roomba 500-series vacuums and the Thames and Kosmos toys, but can understand Pronto IR codes, so simply by putting some custom control html pages on your phone, you can control via browser any device that you have a Pronto IR code for.

Step 4: First-person View Roomba Control: Hardware

The hardware here is very simple: a wooden stand to which the phone attaches with a rubber band, plus the Griffin infrared emitter lying there.

I made one modification to the Griffin emitter: it had clips for attaching to a phone, and I just popped these off. These let the emitter lie flat on the Roomba, with its infrared LEDs shining through a gap in the underside of the stand. If you want to make your own emitter, this may be the time to do that.

The stand is made of scrap softwood.

1. Cut a rectangle the size of your phone (in case if you use a case). That was two quick cuts with a sliding miter saw.

2. Cut a rectangle out of the rectangle to make space for the camera and its flash.

3. Cut legs for the stand. Their length is approximately the width of the phone. Make sure they don't interfere with phone's buttons. In my case, that meant one narrow leg and one that could be wider. I guess I could have made them both narrow.

4. Glue the legs to the stand. Glue them in such a way that if you attach the phone in the correct orientation, it will be well balanced.

5. Attach phone with rubber band to the rectangle in the stand. Then run a rubber band under the Roomba handle to each of the stand's legs, looping the band around each end of the leg. (Or find some other way to attach, e.g., with Velcro.) Place the emitter so it shines through the gap between the legs.

Step 5: First-person Roomba View: Operation

I'll assume you're controlling the Roomba with a first-person view via the web with a laptop on a WiFi network. (If you don't have WiFi, see Step 3.) It should work similarly for controlling on a tablet. I don't recommend doing this with a non-giant phone, as you'll need room on screen for both controls and video.

1. On your Roomba-mounted phone, install IP Webcam (or equivalent webcam server) and my IR Server.

2. Run IP Webcam or other webcam server and set it up as in Step 2.

3. But now there is one more thing to do: Make sure your webcam server software runs in the background. In IP Webcam, you click on "Actions" and choose "Run in background".

4. Launch IR Server.

5. Plug the infrared emitter and then press the on-off button to activate the IR Server. (The order here is important!)

6. IR Server will give you the address to connect to on your web browser. Connect to that address on your browser.

7. Click on "Controllers plus webcam". You should now have a split screen with a choice of controllers (currently Thames and Kosmos, and Roomba) and a camera. Choose Roomba and in the camera screen turn on Flash or Browser viewing.

8. Turn on the Roomba with the physical button (infrared power-up doesn't seem to always work).

9. Control it to your heart's content. Note that the standard turn and forward buttons need to be held down (just like on a remote control) for the Roomba to run, and the Roomba will stop when you release them. If you just click them, probably nothing will happen, as a stop will be sent very quickly after a start. Buttons like "Spot" need only be clicked.

Note that the vacuum will be on whenever the Roomba moves. (Future plans include connecting directly to the Roomba's serial port which will let one have better control over this and other things.)

1. Repeat without the infrared emitter and make sure the phone is producing sounds when buttons are pressed in the browser.

2. Make sure that the emitter was plugged in before IR Server was activated. If unsure, just turn off IR Server and turn it back on.

3. Use another phone's camera to check if the emitter is producing infrared signals. Phone cameras tend to show infrared (better digital cameras block infrared so they can't be used for this).

4. Experiment with IR Server's settings: Press the menu key and choose "Settings". If you're using the Griffin emitter, the default settings should work (though performance may be improved by setting PCM-8). If you're using another emitter, you might adjust the stereo settings. You might try the 90 degree out of phase option, for instance.

Step 6: First-person Thames and Kosmos View: Hardware

We built the Thames and Kosmos bulldozer model. Then I modified it by moving the battery pack forward a little to make space for the phone. The phone just sits there. Finally, I rubber-banded the infrared emitter to the battery compartment.

The only tricky thing is making sure that none of the cables get in the way to the camera.

Step 7: First Person Thames and Kosmos View: Operation

Use the same method as for the Roomba, except now choosing the Roomba controllers. You'll notice a strange "Left/Right Forward" (and "Left/Right Backward") control. This is equivalent to pressing both the left and right forward buttons together on the Thames and Kosmos remote control.

I've noticed that occasionally in testing the Thames and Kosmos stopped responding and needed to be "primed" with its own remote control, or to have the batteries pulled and put back in.

Step 8: Advanced: Adding Support for Your Own Devices

If you have new devices to support (other robots, TVs, etc.), you'll need to know the IR codes for your device. IR Server supports Pronto format codes so if you have Pronto-compatible capture hardware, you can just use that. I myself decoded the Thames and Kosmos codes using a Brainlink device (SurplusShed had them on sale).

What you need to have in hand is either a Pronto-compatible hex sequence, or else the following information:

• carrier frequency (usually around 38 khz)
• burst sequence for each function: this is a sequence of on and off times in microseconds
• amount of time in microseconds to pause between re-sending a burst sequence when emulating a button being held down (usually 20000-100000 microseconds)

Using the above information, you can generate IR Server compatible control queries. For Pronto codes, the format is:

?ir=repeatmode:pronto:xxxx+xxxx+xxxx+xxxx+...

where repeatmode is one of:

• infinite : repeat until a stop code is sent
• count=x : repeat x times (x is a positive integer)
• time=x : repeat x microseconds (x is a positive integer)
• once : play once

and xxxx+xxxx+xxxx+xxxx+... is the Pronto hex code with spaces replaced by plus signs for web use. For instance, the web specifies the Pronto code for Roomba's clean function as

0000 0069 0000 0008 0071 0027 0023 0071 0023 0071 0023 0071 0071 0027 0023 0071 0023 0071 0023 030B

You can then generate the IR Server query:

?ir=time=500000:pronto:0000+0069+0000+0008+0071+0027+0023+0071+0023+0071+0023+0071+0071+0027+0023+0071+0023+0071+0023+030B

This plays the clean code for half a second (500000 microseconds).

If you have the raw data, instead your format is:

?ir=repeatmode:raw:repeatpause:carrier:xxx:xxx:xxx:...

Here repeatmode is the same as for the Pronto codes, repeatpause is a non-negative integer specifying the number of microseconds to pause between repeats (ignored if repeatmode is once)--I recommend trying 20000 as a start, and carrier is the carrier frequency in Hz. Finally, the burst sequence xxx:xxx:xxx:... is a sequence of numbers specifying alternating on and off times in microseconds. Thus:

?ir=count=12:raw:20000:38000:400:200:400:200:300:150:100

requests that the burst sequence be emitted twelve times with a 38000 Hz carrier, with a 20000 microsecond pause in between. The sequence then is 400 microseconds on, 200 off, 400 on, 200 off, 300 on, 150 off, 100 on.

There is also a special query:

?ir=stop

which stops a previous play command with infinite, count or time repeatmode.

Before doing any html editing, I recommend testing the codes by setting up IR Server connected to an emitter, and manually emitting the codes via a web browser. For instance, if IR Server is at 168.1.1.49:7080, you can emit my sample raw code by going to:

http://168.1.1.49:7080/ir.html?ir=count=12:raw:20000:38000:400:200:400:200:300:150:100

Note that it doesn't matter which web page you put for "ir.html"--the code will still be emitted. For greatest efficiency, you can do "empty.html".

Once you've worked out the query codes for your device functions, you need to create an html file control it. Once you've installed IR Server, its html files are in the mobi.omegacentauri.irserver directory of your phone's internal storage. You will find a bunch of files that you're free to edit there (if you want to refresh them with the official versions, press Menu and Overwrite in IR Server). The roomba.html file is a good start. Copy that to a new file (e.g., xyzzy.html) and edit it to have the kind of layout you want your device to have, and to edit the codes.

The basic idea is that when IR Server gets a request with one of the ?ir= query codes, it decodes the code and plays it. The request can be appended to any html file, but the empty.html file is most efficient. Also, for efficiency, instead of issuing a GET request, you can just issue a HEAD request.

The roomba.html file contains a bit of javascript designed to emit the requests upon appropriate button presses. There are two kinds of buttons. Repeating buttons play a code repeatedly until one lets go of the button. These use an infinite mode, and have both ontouchstart and onmousedown code to work on both touch and mouse based browsers. (While I'm an experienced C programmer, and a somewhat experience java programmer, I am completely new to javascript, so I may not be doing things the best way.) Non-repeat buttons play a code for a set amount of time so it doesn't get missed, and do so via onclick. There are different javascript functions to call for these.

You will notice that there are special simplified ?ir requests for Roomba and Thames and Kosmos codes. For these, only a single decimal number needs to be specified instead of a burst sequence, and it gets encoded according to the Roomba and Thames and Kosmos protocols (see the IRCommand class in the IR Server source for how this works).

Put your new html file in the mobi.omegacentauri.irserver directory of your phone's internal storage. You can access it directly by name, or you can add a link to it in the controllers.html file.