Introduction: Remote Controlled Webcam Using Arduino, SensorMonkey, JQuery and Justin.tv
Web-enable your interactive sensors over desktop, smartphone and tablet devices.
This tutorial describes in detail how to use the free SensorMonkey service to remotely control a pan and tilt webcam attached to an Arduino using nothing more than a simple webpage. The webpage can be viewed on any desktop, smartphone or tablet device with a compatible web-browser. I use the jQuery UI library to provide interactive pan and tilt controllers and Justin.tv to provide the live streaming audio/video output captured by the webcam. No server-side coding or Ethernet shield is required.
In a previous tutorial, I described in detail how to use SensorMonkey to push real-time sensor data from an Arduino to a webpage for visualization using Processing.js (Disclosure: I co-founded the company developing SensorMonkey). I used an accelerometer to sample motion data and published it live over the Internet for people to view its output. The resulting webpage could be embedded into an external website to act as a real-time streaming widget. Communication was uni-directional; the data flowed from the Arduino through SensorMonkey and into the webpage. This tutorial demonstrates the opposite; namely, the data flows from the webpage through SensorMonkey and into the Arduino.
Important! Although not strictly necessary, it is recommended that you have read the previous tutorial in this series (at the very least, you should be familiar with the basic workflow used to connect an Arduino to SensorMonkey).
Step 1: Gathering Materials
Hardware:
- Arduino (I use an Uno but older boards such as a Duemilanove will work fine)
- USB cable to connect Arduino to host computer
- 2 x servo motors - one for pan and one for tilt (I use Hitec HS-422 Deluxe servos)
- Mounting brackets to construct pan and tilt assembly using servos (I use a Lynx B-Pan/Tilt kit)
- A base of some kind to stabilize your pan and tilt assembly (I use a cheap plastic mini-vice!)
- Webcam (I use a Microsoft LifeCam VX-3000 but any will do as long as it's compatible with your OS)
- External power supply to provide current to servos (I use rechargeable Ni-MH 9V batteries)
- 9V to barrel jack adapter to connect external power supply to Arduino
- Assorted wires to connect pan and tilt assembly to Arduino
- Breadboard
- Arduino development environment (http://www.arduino.cc)
- Free account on SensorMonkey.com (login with your existing Facebook account)
- Free account on Justin.tv (for live audio/video streaming)
- Bloom (for Windows users) orSensorMonkeySerialNet (for non-Windows users)
- jQuery UI
The 9V batteries don't last long, so if you want to setup your webcam for an extended period of time you'll need to hook it up to a persistent external power supply such as a wall-mounted DC adapter or a high-power density LiPo battery.
Alternatively, you can power one of the 2 servos using another Arduino connected to a different USB port.
Step 2: Mount Webcam, Connect Arduino and Upload Sketch
First, I build the pan and tilt assembly by attaching the servos to the mounting brackets as described here (if you're using different components, please follow the relevant assembly guide(s) for your particular servos and mounting brackets instead).
Next, I attach the webcam to the top of the tilt servo's mounting bracket. In my case, I simply had to remove (unscrew) the universal attachment base from the bottom of the webcam and screw the device into one of the holes in the mounting bracket. Depending on your webcam, you may need to attach it by some other means (you can always use sticky tape if all else fails!).
To stabilize the whole assembly, I place it into the plastic mini-vice and fix the vice to a flat surface (i.e. the top of my desk). Again, depending on your components, you may have other requirements. As long as the webcam can pan and tilt without falling over that's all that matters.
From here, I wire the servos to the Arduino as shown in the pictures and the circuit diagram (made using Fritzing). The tilt servo is connected to analog pin 0, while the pan servo is connected to analog pin 5. The Arduino is connected to the host computer using the USB cable and powered using the external power supply through the built-in barrel jack adapter.
Finally, to control the servos, I upload the following sketch to the Arduino's microcontroller using the development environment:
#include <Servo.h>
Servo pan, tilt;
void setup() {
pan.attach( A5 );
tilt.attach( A0 );
Serial.begin( 9600 ); // Open the serial port.
}
void loop() {
if( Serial.available() ) {
byte b = Serial.read();
// Map high 4 bits of incoming byte to pan rotation in degrees.
pan.write( map( b >> 4, 0, 15, 0, 180 ) );
delay( 15 );
// Map low 4 bits of incoming byte to tilt rotation in degrees.
tilt.write( map( b & 0x0F, 0, 15, 0, 180 ) );
delay( 15 );
}
}
The sketch is very basic. It opens the serial port and reads bytes one at a time. Each byte is assumed to contain a pan and tilt rotation pair; the high 4 bits are the pan rotation (0 to 15 inclusive) and the low 4 bits are the tilt rotation (0 to 15 inclusive). This gives 16 different levels (i.e. 24) to choose from with respect to each dimension of motion and makes it easy to encode the webcam's position using hexadecimal character pairs. Each servo has a range of 0 to 180 degrees. So, for example, a hexadecimal character pair of 7A means 7/15 x 180 (84 degrees) on the pan axis and 10/15 x 180 (120 degrees) on the tilt axis. A hexadecimal character pair of 00 encodes a 0 degree rotation on both pan and tilt axes, while FF encodes a full 180 degree rotation on both pan and tilt axes. The mapping for each character is shown below:
0 : 0 degrees
1 : 12 degrees
2 : 24 degrees
3 : 36 degrees
4 : 48 degrees
5 : 60 degrees
6 : 72 degrees
7 : 84 degrees
8 : 96 degrees
9 : 108 degrees
A : 120 degrees
B : 132 degrees
C : 144 degrees
D : 156 degrees
E : 168 degrees
F : 180 degrees
If I needed fine-grained control of the servos' motion, I could encode the pan and tilt rotations as separate bytes. In this case, however, using a single byte only is an efficient means of encoding the co-ordinate system for controlling the two servos and provides adequate motion range for a simple webcam.
Step 3: Download and Install Bloom (or SensorMonkeySerialNet)
- constructed a pan and tilt assembly using 2 servo motors
- mounted a webcam to the pan and tilt assembly
- wired the pan and tilt assembly to an Arduino
- connected the Arduino to a host computer over USB
Bloom is a serial port to TCP/IP socket redirector. It listens for incoming connections on a user-specified TCP/IP port. When a connection is accepted, Bloom will open the serial port and transfer data back and forth between the TCP/IP socket and the serial port, allowing SensorMonkey to connect to the Arduino as if it were a networked device with an Ethernet shield. I use the following settings:
- TCP/IP port: 20000
- Polling frequency: 50
- Serial port: COM8
- Baud rate: 9600
- Data bits: 8
- Parity bit: None
- Stop bits: 1
- Flow control: None
- port: 20000
- pollingFreq: 50
- baudRate: 9600
- waitTime: 1000
Step 4: Login to SensorMonkey and Publish Stream for Controlling Arduino
You can login to SensorMonkey using your existing Facebook account by clicking the 'Login with Facebook' button in the top-right corner of the page.
You will be prompted to grant permission for the SensorMonkey application to access your Facebook account. Once you have done so, you will be assigned a personal namespace (a streaming 'sandbox' for your sensors) as well as public and private keys to access your namespace from within a webpage. You will need your private key for Step 6. You can find it by clicking the 'Namespaces' link at the top of the page.
Once logged in, you can access the web-based control panel through the 'Sensors' link at the top of the page. The control panel is where you will connect to the Arduino and publish its stream live over the Internet.
After logging into SensorMonkey and opening my control panel, I'm going to add an entry for the Arduino named "My Webcam". By clicking on the newly added entry, I can configure the connection parameters; namely, the IP address and port number where the device can be found.
Recall from Step 3 that I am using Bloom (or SensorMonkeySerialNet) to map the Arduino's serial port to TCP/IP port 20000 on my local machine. So, I enter a port number of 20000 and an IP address of 127.0.0.1 (the local loopback address). I'm not reading any data from the Arduino, so I can use the default format description file provided by the control panel.
After clicking 'Connect', I navigate to the 'Control' tab where I can test my pan and tilt assembly by sending commands to the Arduino. By prefixing the commands with a # symbol, SensorMonkey will interpret the text as hexadecimal character pairs (i.e. binary octets). So, for example, I can instruct the pan and tilt assembly to assume a rotation of 180 degrees on both axes by typing #FF into the text field and pressing return on my keyboard (or clicking the 'Send Text' button). Try out the following combinations to test your pan and tilt assembly (be careful not to exceed the practical rotation range of your servo motors):
#08 : Pan 0 degrees, Tilt 96 degrees
#0F : Pan 0 degrees, Tilt 180 degrees
#FF : Pan 180 degrees, Tilt 180 degrees
#F8 : Pan 180 degrees, Tilt 96 degrees
Finally, after testing my pan and tilt assembly, I navigate to the 'Stream' tab where I can publish the stream for controlling the Arduino live over the Internet. I'm required to select at least one variable when streaming (even if I don't actually use it) so I select the default variable ('Unsigned 8-bit Variable'), choose a stream type of 'Private', and click 'Publish'. The stream must be made private so as to allow remote clients to write to it.
In Step 6, I will write a simple HTML webpage to connect to my namespace, subscribe to my stream, and allow me to send commands to the Arduino to control the pan and tilt assembly using interactive sliders.
Step 5: Login to Justin.tv and Publish Audio/Video Stream From Webcam
Justin.tv is a free online service for broadcasting live audio/video streams. After creating an account, I can publish my live webcam feed by clicking the 'Go Live!' button in the top-right corner of the page. I allow access to my webcam when asked and click the 'Start' button to begin broadcasting. Once my channel is live, I can embed the output in a webpage using some simple HTML code (see the next step).
Important! If you have more than one webcam make sure to select the one that is mounted to the pan and tilt assembly. You can do this through the channel options when broadcasting your stream.
Step 6: Remotely Control Webcam Using JQuery UI
In the final part of this tutorial, I'm going to combine the live streams from SensorMonkey and Justin.tv to create a simple webpage that can be used to remotely control my webcam. I have downloaded the latest jQuery UI library (1.8.21 at the time of writing) and placed it in the same directory as the webpage (along with the associated CSS and image files for my chosen theme - UI lightness - see screenshot). You'll need to edit the code below and save it as 'Webcam.html':
(Important! You must replace YOUR_NAMESPACE and YOUR_PRIVATE_KEY in the code below with those assigned to you when you login to SensorMonkey. You'll also need to replace YOUR_CHANNEL with the name of your Justin.tv channel)
--------------------------------------------------------------------------------
<!DOCTYPE html>
<html>
<head>
<title>Remote controlled webcam using Arduino, SensorMonkey, jQuery and Justin.tv</title>
<link rel="stylesheet" type="text/css" href="jquery-ui-1.8.21.custom.css" />
<style type="text/css">
body {
padding: 10px;
}
#container {
margin-bottom: 20px;
}
#webcam {
float: left;
height: 240px;
margin-right: 20px;
width: 320px;
}
#tilt-slider {
float: left;
height: 240px;
margin-right: 10px;
}
#tilt-display {
height: 240px;
line-height: 240px;
}
#pan-slider {
margin-bottom: 10px;
width: 320px;
}
#pan-display {
text-align: center;
width: 320px;
}
.rotation {
color: #F6931F;
font-weight: bold;
}
</style>
<script type="text/javascript" src="jquery-1.7.2.min.js"></script>
<script type="text/javascript" src="jquery-ui-1.8.21.custom.min.js"></script>
<script type="text/javascript" src="https://sensormonkey.eeng.nuim.ie/socket.io/socket.io.js"></script>
<script type="text/javascript" src="https://sensormonkey.eeng.nuim.ie/js/client.min.js"></script>
</head>
<body>
<div id="container">
<div id="webcam">
<object type="application/x-shockwave-flash" data="http://www.justin.tv/widgets/live_embed_player.swf?channel=YOUR_CHANNEL" id="live_embed_player_flash" height="240" width="320" bgcolor="#000000">
<param name="allowFullScreen" value="true" />
<param name="allowScriptAccess" value="always" />
<param name="allowNetworking" value="all" />
<param name="movie" value="http://www.justin.tv/widgets/live_embed_player.swf" />
<param name="flashvars" value="hostname=www.justin.tv&channel=YOUR_CHANNEL&auto_play=true&start_volume=25" />
</object>
</div>
<div id="tilt-slider"></div>
<div id="tilt-display">Tilt: <span class="rotation">96</span></div>
</div>
<div id="pan-slider"></div>
<div id="pan-display">Pan: <span class="rotation">96</span></div>
<script type="text/javascript">
// Converts an integer (or a string representation of one) to a hexadecimal character (0-9A-F).
function toHex( i ) {
return parseInt( i ).toString( 16 ).toUpperCase();
}
$( function() {
// Create tilt slider.
$( "#tilt-slider" ).slider( {
range : "min",
orientation : "vertical",
value : 8,
min : 0,
max : 15,
step : 1,
slide : function( event, ui ) {
// Update UI.
$( "#tilt-display .rotation" ).html( 180 * ui.value / 15 );
// Calculate combined pan/tilt rotation angles and send to the stream publisher
// as a hexadecimal character pair. Pan is high 4 bits; tilt is low 4 bits. By
// prefixing with '#', we tell SensorMonkey to interpret as binary data.
var pan = toHex( $( "#pan-slider" ).slider( "value" ) );
var tilt = toHex( ui.value );
client.deliverToStreamPublisher( "/private/My Webcam", "#" + pan + tilt );
}
} );
// Create pan slider.
$( "#pan-slider" ).slider( {
range : "min",
value : 8,
min : 0,
max : 15,
step : 1,
slide : function( event, ui ) {
// Update UI.
$( "#pan-display .rotation" ).html( 180 * ui.value / 15 );
// Calculate combined pan/tilt rotation angles and send to the stream publisher
// as a hexadecimal character pair. Pan is high 4 bits; tilt is low 4 bits. By
// prefixing with '#', we tell SensorMonkey to interpret as binary data.
var pan = toHex( ui.value );
var tilt = toHex( $( "#tilt-slider" ).slider( "value" ) );
client.deliverToStreamPublisher( "/private/My Webcam", "#" + pan + tilt );
}
} );
// 1. Connect to SensorMonkey
// 2. Join namespace
// 3. Subscribe to stream
var client = new SensorMonkey.Client( "https://sensormonkey.eeng.nuim.ie" );
client.on( "connect", function() {
client.joinNamespace( "YOUR_NAMESPACE", "YOUR_PRIVATE_KEY", function( e ) {
if( e ) {
alert( "Failed to join namespace: " + e );
return;
}
client.subscribeToStream( "/private/My Webcam", function( e ) {
if( e ) {
alert( "Failed to subscribe to stream: " + e );
return;
}
} );
} );
client.on( "disconnect", function() {
alert( "Client has been disconnected!" );
} );
} );
} );
</script>
</body>
</html>
--------------------------------------------------------------------------------
The code inside of the <object></object> tags is used to embed the live Justin.tv stream into the webpage. You'll need to replace each instance of YOUR_CHANNEL in the code between these tags with the name of your channel. The stream won't display on iOS devices (iPhone, iPad etc.) but will work on Android smartphones and tablets, as shown in the photos.
I use jQuery UI to create a horizontal pan slider and a vertical tilt slider to control the orientation of the webcam. When one of the sliders is moved to a new position, the code calculates a combined pan/tilt rotation, encodes it as a hexadecimal character pair (as described in Step 2 and Step 4) and sends it through SensorMonkey to the Arduino controlling the servo motors.
Finally, the workflow for connecting to SensorMonkey is very simple (don't forget to replace YOUR_NAMESPACE and YOUR_PRIVATE_KEY in the code above):
- Import client
- Connect to SensorMonkey
- Join namespace
- Subscribe to stream
Once subscribed, I can simply call client.deliverToStreamPublisher() to send data directly to the Arduino through the SensorMonkey service.
That's it! I can now remotely control my webcam in real-time using a combination of Arduino, SensorMonkey, jQuery and Justin.tv. I can upload the webpage to a public webserver and access it from anywhere on any device with a HTML5/Flash compatible web-browser. See the next step for suggestions on improving the implementation described so far.
Step 7: Improvements and Enhancements
- Touchscreen UI: The standard jQuery UI library is not optimized for use on touchscreen devices. As such, the sliders can be a little unwieldy and difficult to manipulate properly on mobile devices. jQuery Mobile can be used instead to provide more intuitive and easy to use UI controls across all popular mobile device platforms.
- Synchronization of Multiple Remote Clients: Currently, the pan and tilt sliders are not synchronized among multiple remote clients. In other words, when one client moves the webcam the change is not reflected in the sliders of the other clients. One way to accomplish this would be to have the Arduino broadcast the current rotations of the servos whenever they are updated. You could then listen for 'publish' and 'bulkPublish' events in the JavaScript code and synchronize the sliders each time an update is received.
- High Resolution Video Encoding: The default encoder used by Justin.tv is not very good. To improve the quality, you can use Wirecast or Flash Media Encoder to produce a higher resolution stream that can be broadcast through Justin.tv instead.
- Alternative/Custom Video Streaming Services: If Justin.tv is not to your liking, there are other free alternatives; Livestream, Ustream.tv and Bambuser to name three of the more popular ones. If you're feeling adventurous, you can setup your own instance of Wowza Media Server on Amazon EC2 to stream your live audio/video feeds.
- Mouse Control: Instead of sliders, you can use the position of the mouse to control the orientation of the webcam by mapping the co-ordinates of the onscreen cursor to pan and tilt rotations. You can then encode the rotation angles as hexadecimal character pairs before sending them to the Arduino through SensorMonkey.
- Remote Control of Additional Actuators: As well as sending commands to control the servos mounted in the pan and tilt assembly, you can send commands to control additional actuators wired to the Arduino. Just define your commands and have the Arduino's firmware parse the incoming bytes from the serial port to identify the types and arguments. You can provide buttons on your webpage to turn LEDs and switches on/off or custom control elements for driving other servos.
- Fine-grained Motion Control: Instead of quantizing the pan and tilt rotations to fit together in a single byte, you could encode them separately and allow the full range of servo motion to be controlled using the sliders (i.e. 0 to 180 degrees in single degree increments).
