Mimbo - a Friendly Robot
Intro: Mimbo - a Friendly Robot
Mimbo is a friendly robot who mimics your emotions. He has a cardboard box body, an iPhone for a face, and a Processing sketch for a brain.
STEP 1: Build the Body
Mimbo's body is made out of a single sheet of cardboard.
The best way to make the body is to first glue the pattern on the cardboard. Then, using an exacto knife, cut out the pattern. Score the fold lines lightly and valley fold such that the paper pattern is now on the inside and nice clean cardboard is on the exterior.
The files provided are designed for an iPhone 3GS but also work with an iPhone 4/4s with little fuss.
To Design Your Own:
- If you're a student, use Solidworks or Autodesk Inventor to turn your 3D model into a flat drawing with the sheet metal tools.
- Otherwise check out Pepakura
The best way to make the body is to first glue the pattern on the cardboard. Then, using an exacto knife, cut out the pattern. Score the fold lines lightly and valley fold such that the paper pattern is now on the inside and nice clean cardboard is on the exterior.
The files provided are designed for an iPhone 3GS but also work with an iPhone 4/4s with little fuss.
To Design Your Own:
- If you're a student, use Solidworks or Autodesk Inventor to turn your 3D model into a flat drawing with the sheet metal tools.
- Otherwise check out Pepakura
STEP 2: Create the TouchOSC Interface
First, go and download the TouchOSC Editor and TouchOSC app for your smart phone (both Android and iPhone supported). You can get them both here. (The app is $5 but well worth it in my opinion)
The editor is super intuitive and quite easy to use. To add a feature just right click and select the object. You can edit the color, position, name, value range, and a number of other parameters by just selecting the object and editing the respective values.
Mimbo's face is composed of 6 features: 4 LED's (two eyes and two pupils), 1 empty label box (eyebrow/eyelid), and 1 multi-fader (mouth). All of these are changed and controlled programmatically in the processing code.
Download my interface or create your own and then just hit the "Sync" button and update the layout on your smartphone. You may need to specify the IP address of your computer but the instructions on the TouchOSC download page are pretty clear.
The editor is super intuitive and quite easy to use. To add a feature just right click and select the object. You can edit the color, position, name, value range, and a number of other parameters by just selecting the object and editing the respective values.
Mimbo's face is composed of 6 features: 4 LED's (two eyes and two pupils), 1 empty label box (eyebrow/eyelid), and 1 multi-fader (mouth). All of these are changed and controlled programmatically in the processing code.
Download my interface or create your own and then just hit the "Sync" button and update the layout on your smartphone. You may need to specify the IP address of your computer but the instructions on the TouchOSC download page are pretty clear.
STEP 3: Control TouchOSC Elements From Processing
To talk to TouchOSC using the Open Sound Control protocol you'll first need to download and install the oscP5 library for Processing here.
Once you figure out the basic structure of messages, OSC is a really simple and effective way of talking to all kinds of multimedia devices.
In the setup you'll need to do two things: initialize oscP5 and tell it what port to listen to for incoming messages, as well as declare a remote address (in this case the IP address of the smartphone) and the outgoing port. You can find these two values in the OSC app.
oscP5 = new OscP5(this,8000); // Start oscP5, listening for incoming messages at port 8000
myRemoteLocation = new NetAddress("10.0.1.3",9000); // IP address, outgoing port (taken from the TouchOSC app)
An OSC signal is composed of a message and a value. Let's look at the simple case of turning a TouchOSC LED on and off:
OSCMessage myMessage = new OscMessage("/1/led1"); // <--- where /1 is the first page and /led1 is led1
myMessage.add(1); // <-- assign a value to the message (in this case turn the led on)
oscP5.send(myMessage, myRemoteAddress); // <-- send the message to the remote address
To dim the LED to half of its maximum brightness you just need to send another message exactly as above but with 0.5 instead 1 in the myMessage.add(__).
The oscP5Message example pictured provides a good baseline to work from.
Once you figure out the basic structure of messages, OSC is a really simple and effective way of talking to all kinds of multimedia devices.
In the setup you'll need to do two things: initialize oscP5 and tell it what port to listen to for incoming messages, as well as declare a remote address (in this case the IP address of the smartphone) and the outgoing port. You can find these two values in the OSC app.
oscP5 = new OscP5(this,8000); // Start oscP5, listening for incoming messages at port 8000
myRemoteLocation = new NetAddress("10.0.1.3",9000); // IP address, outgoing port (taken from the TouchOSC app)
An OSC signal is composed of a message and a value. Let's look at the simple case of turning a TouchOSC LED on and off:
OSCMessage myMessage = new OscMessage("/1/led1"); // <--- where /1 is the first page and /led1 is led1
myMessage.add(1); // <-- assign a value to the message (in this case turn the led on)
oscP5.send(myMessage, myRemoteAddress); // <-- send the message to the remote address
To dim the LED to half of its maximum brightness you just need to send another message exactly as above but with 0.5 instead 1 in the myMessage.add(__).
The oscP5Message example pictured provides a good baseline to work from.
STEP 4: Receive OSC Messages in Processing
Mimbo utilizes the accelerometer feature in TouchOSC to send it's acceleration sensor values back to the processing script.
In order to receive OSC messages, an interrupt/event handler method is required. The simplest way to debug OSC event handling is to simply printout whatever is received using code like this:
void oscEvent(OscMessage theOscMessage)
{
println(theOscMessage.toString());
}
To get the value(s) of a message employ the following command in the oscEvent function:
float myVariable = theOscMessage.get(0).floatValue();
In the case of Mimbo, the OSC Event handler looks like this:
void oscEvent(OscMessage theOscMessage)
{
// This runs whenever there is a new OSC message
String addr = theOscMessage.addrPattern(); // Creates a string out of the OSC message
if (addr.indexOf("/accxyz") != -1) {
last_accx = accx;
last_accy = accy;
accx = theOscMessage.get(0).floatValue();
accy = theOscMessage.get(1).floatValue();
accz = theOscMessage.get(2).floatValue();
}
In order to receive OSC messages, an interrupt/event handler method is required. The simplest way to debug OSC event handling is to simply printout whatever is received using code like this:
void oscEvent(OscMessage theOscMessage)
{
println(theOscMessage.toString());
}
To get the value(s) of a message employ the following command in the oscEvent function:
float myVariable = theOscMessage.get(0).floatValue();
In the case of Mimbo, the OSC Event handler looks like this:
void oscEvent(OscMessage theOscMessage)
{
// This runs whenever there is a new OSC message
String addr = theOscMessage.addrPattern(); // Creates a string out of the OSC message
if (addr.indexOf("/accxyz") != -1) {
last_accx = accx;
last_accy = accy;
accx = theOscMessage.get(0).floatValue();
accy = theOscMessage.get(1).floatValue();
accz = theOscMessage.get(2).floatValue();
}
STEP 5: Putting Things Together
Once you've got the basics of sending and receiving OSC messages with processing and a TouchOSC interface on your phone what do you do? You put them together and make Mimbo!
TouchOSC actually allows you to control far more than just the value of the element. You can also (in most cases) control it's color, position, size, etc...
So along with sending a message to "/1/led1" with the LED brightness, we send a message to "/1/led1/position/y" to control it's y-position on the screen.
All of the controls are pretty well documented on this page.
After a whole bunch of tweaking you end up with this code and cute little robot face on your phone.
TouchOSC actually allows you to control far more than just the value of the element. You can also (in most cases) control it's color, position, size, etc...
So along with sending a message to "/1/led1" with the LED brightness, we send a message to "/1/led1/position/y" to control it's y-position on the screen.
All of the controls are pretty well documented on this page.
After a whole bunch of tweaking you end up with this code and cute little robot face on your phone.
STEP 6: Adding FaceTracking
To take Mimbo to the next level you need have him to track your face and mimic your facial expressions.
Fortunately a lot of the really complex facial tracking problems have been solved for us already. Yay open source! :)
FaceOSC is a pretty simple application (unfortunately OSX 10.6 only) that tracks your face using your webcam and sends OSC messages to a specified port. We can then either read-in these messages directly in processing (on a new incoming port) or re-route them through a program like Osculator to our standard incoming port in Processing. The benefit of routing them through Osculator is that you can very quickly plot whatever signal you want and figure out it's range of values.
Now it's simply a matter of mapping facial expressions (ie. mouth width, eyebrow height) to Mimbo's expressions.
Fortunately a lot of the really complex facial tracking problems have been solved for us already. Yay open source! :)
FaceOSC is a pretty simple application (unfortunately OSX 10.6 only) that tracks your face using your webcam and sends OSC messages to a specified port. We can then either read-in these messages directly in processing (on a new incoming port) or re-route them through a program like Osculator to our standard incoming port in Processing. The benefit of routing them through Osculator is that you can very quickly plot whatever signal you want and figure out it's range of values.
Now it's simply a matter of mapping facial expressions (ie. mouth width, eyebrow height) to Mimbo's expressions.
32 Comments
adianand 10 years ago
adianand
I do not have a iphone so what can i use and send all the Touch osc downloads and other downloads to me on anand_hn@yahoo.com please.i love this project and i love robots
instrumata 11 years ago
wbautista1 11 years ago
wbautista1 11 years ago
ynze 12 years ago
Inspired in your Mimbo, roy-t and I built a Windows-based version. We named it Wimbo :-)
We're currently working on an instructable for it. In the mean time: here are some pics...
it runs on a PC with two displays. Roy-t is the programming guy, he wrote the OSC alternative. I did the hardware.
Thanks for sharing!
sewonist 12 years ago
It's really nice work.
so I just follow you. I've made like this one.
You can watch the video on my blog (http://sewonist.com/?p=2776)
Cheers
Flyinseamnky 12 years ago
Cmac678 12 years ago
wkl 12 years ago
Cmac678 12 years ago
I can get the code running to my phone.
Thank You
wkl 12 years ago
Is FaceOSC running properly and showing a mesh on your face? If it's running and can't find you it should say "searching for face..." in the top left corner of the video feed. Otherwise, if it's found you, it'll have a frame rate (some number that usually varies between 30 and 130). If all of that seems good then the problem is on the osculator side.
On the osculator side.... is it running? There's a big green pause/start button in the top left corner. If it's running and FaceOSC seems to be working but /found is still 0 then it could be that FaceOSC is sending data to the wrong OSC port or osculator is listening at the wrong incoming port. The default value in the osculator .oscd I provided is 8338. To check the port that FaceOSC is sending to right click on the FaceOSC application and select "show package contents." Then, navigate to /contents/data and open settings.xml. The osc output settings should be set to localhost and 8338.
Let me know if that helps or not...
Cmac678 12 years ago
wkl 12 years ago
It should look like this...
....
} else if (addr.indexOf("/pose/position/0") != -1) {
faceXPos = theOscMessage.get(0).floatValue();
} else if (addr.indexOf("/pose/position/1") != -1) {
faceYPos = theOscMessage.get(0).floatValue();
}
println(mouthWidth);
}
Emiliano Valencia 12 years ago
wkl 12 years ago
Under "Will's iPhone" or "so and so's iPod Touch" double click in the empty field. A cursor should appear and you just need to type: localhost:8000. Then ensure that it is selected with the radio button.
jasechong 12 years ago
1) In Osculator, click the Parameters button
2) Change to the OSC Routing tab
3) Check that localhost:8000 is selected, or add it if not in there. I found mine was routing elsewhere and thus Processing and the Mimbo pde script didn't get any events
Emiliano Valencia 12 years ago
jimkiller 12 years ago
wkl 12 years ago
When you look in your sketchbook do you see the oscP5 library there?
(File->Sketchbook->libraries->oscP5).
If so, open one of the examples and see if that works....
jimkiller 12 years ago