Mimbo - a Friendly Robot

Mimbo is a friendly robot who mimics your emotions. He has a cardboard box body, an iPhone for a face, and a Processing sketch for a brain.

Step 1: Build the Body

Mimbo's body is made out of a single sheet of cardboard.

The best way to make the body is to first glue the pattern on the cardboard. Then, using an exacto knife, cut out the pattern. Score the fold lines lightly and valley fold such that the paper pattern is now on the inside and nice clean cardboard is on the exterior.

The files provided are designed for an iPhone 3GS but also work with an iPhone 4/4s with little fuss.

To Design Your Own:
- If you're a student, use Solidworks or Autodesk Inventor to turn your 3D model into a flat drawing with the sheet metal tools.
- Otherwise check out Pepakura

Step 2: Create the TouchOSC Interface

First, go and download the TouchOSC Editor and TouchOSC app for your smart phone (both Android and iPhone supported). You can get them both here. (The app is $5 but well worth it in my opinion)

The editor is super intuitive and quite easy to use. To add a feature just right click and select the object. You can edit the color, position, name, value range, and a number of other parameters by just selecting the object and editing the respective values.

Mimbo's face is composed of 6 features: 4 LED's (two eyes and two pupils), 1 empty label box (eyebrow/eyelid), and 1 multi-fader (mouth). All of these are changed and controlled programmatically in the processing code.

Download my interface or create your own and then just hit the "Sync" button and update the layout on your smartphone. You may need to specify the IP address of your computer but the instructions on the TouchOSC download page are pretty clear.

Step 3: Control TouchOSC Elements From Processing

To talk to TouchOSC using the Open Sound Control protocol you'll first need to download and install the oscP5 library for Processing here.

Once you figure out the basic structure of messages, OSC is a really simple and effective way of talking to all kinds of multimedia devices.

In the setup you'll need to do two things: initialize oscP5 and tell it what port to listen to for incoming messages, as well as declare a remote address (in this case the IP address of the smartphone) and the outgoing port. You can find these two values in the OSC app.

oscP5 = new OscP5(this,8000); // Start oscP5, listening for incoming messages at port 8000
myRemoteLocation = new NetAddress("10.0.1.3",9000); // IP address, outgoing port (taken from the TouchOSC app)

An OSC signal is composed of a message and a value. Let's look at the simple case of turning a TouchOSC LED on and off:

OSCMessage myMessage = new OscMessage("/1/led1"); // <--- where /1 is the first page and /led1 is led1
myMessage.add(1); // <-- assign a value to the message (in this case turn the led on)
oscP5.send(myMessage, myRemoteAddress); // <-- send the message to the remote address

To dim the LED to half of its maximum brightness you just need to send another message exactly as above but with 0.5 instead 1 in the myMessage.add(__).

The oscP5Message example pictured provides a good baseline to work from.

Step 4: Receive OSC Messages in Processing

Mimbo utilizes the accelerometer feature in TouchOSC to send it's acceleration sensor values back to the processing script.

In order to receive OSC messages, an interrupt/event handler method is required. The simplest way to debug OSC event handling is to simply printout whatever is received using code like this:

void oscEvent(OscMessage theOscMessage)
{
     println(theOscMessage.toString());
}

To get the value(s) of a message employ the following command in the oscEvent function:

float myVariable = theOscMessage.get(0).floatValue();

In the case of Mimbo, the OSC Event handler looks like this:

void oscEvent(OscMessage theOscMessage)
{
  // This runs whenever there is a new OSC message
  String addr = theOscMessage.addrPattern(); // Creates a string out of the OSC message
  if (addr.indexOf("/accxyz") != -1) {
    last_accx = accx;
    last_accy = accy;
    accx = theOscMessage.get(0).floatValue();
    accy = theOscMessage.get(1).floatValue();
    accz = theOscMessage.get(2).floatValue();
  }

Step 5: Putting Things Together

Once you've got the basics of sending and receiving OSC messages with processing and a TouchOSC interface on your phone what do you do? You put them together and make Mimbo!

TouchOSC actually allows you to control far more than just the value of the element. You can also (in most cases) control it's color, position, size, etc...

So along with sending a message to "/1/led1" with the LED brightness, we send a message to "/1/led1/position/y" to control it's y-position on the screen.

All of the controls are pretty well documented on this page.

After a whole bunch of tweaking you end up with this code and cute little robot face on your phone.

Step 6: Adding FaceTracking

To take Mimbo to the next level you need have him to track your face and mimic your facial expressions.

Fortunately a lot of the really complex facial tracking problems have been solved for us already. Yay open source! :)

FaceOSC is a pretty simple application (unfortunately OSX 10.6 only) that tracks your face using your webcam and sends OSC messages to a specified port. We can then either read-in these messages directly in processing (on a new incoming port) or re-route them through a program like Osculator to our standard incoming port in Processing. The benefit of routing them through Osculator is that you can very quickly plot whatever signal you want and figure out it's range of values.

Now it's simply a matter of mapping facial expressions (ie. mouth width, eyebrow height) to Mimbo's expressions.

Comments

[adianand]

adianand

I do not have a iphone so what can i use and send all the Touch osc downloads and other downloads to me on [e-mail] please.i love this project and i love robots

[instrumata]

I wish to make one, but I don't have Mac. A complete beginner, not understand the coding, but definitely will try once I understand it. Thank for sharing this. Waiting for Windows-based version too. :)

[wbautista1]

This is absolutely one way of being creative and saving the mother earth, let us make the most out of our junk, let us recycle.

[wbautista1]

This just happen to be to be one of the most useful recycling ideas I ever saw on the net. This is useful.

[ynze]

Hi wkl,

Inspired in your Mimbo, roy-t and I built a Windows-based version. We named it Wimbo :-)

We're currently working on an instructable for it. In the mean time: here are some pics...

it runs on a PC with two displays. Roy-t is the programming guy, he wrote the OSC alternative. I did the hardware.

Thanks for sharing!

[sewonist]

Hi Guy

It's really nice work.
so I just follow you. I've made like this one.
You can watch the video on my blog (http://sewonist.com/?p=2776)

Cheers

[Flyinseamnky]

Reminds me of the robot head from Lex!

[Cmac678]

What do I do with the .pde files

[wkl]

Open them in Processing and hit run. (You can download at http://processing.org/) Oh and be sure to change the IP address to that of your computer and phone in the code as well.

[Cmac678]

Thanks for your help. I got everything up and running, but when I open the .oscd file and look at the /found it shows 0 and the other values don't change, also the eyes are fading in and out on my phone. What can I do? Sorry, this is the first time I am doing something like this, its really cool though.
I can get the code running to my phone.

Thank You

[wkl]

It's great that you have the eyes fading in and out. It means the processing sketch is communicating with the phone... he's just asleep because he doesn't see anyone.

Is FaceOSC running properly and showing a mesh on your face? If it's running and can't find you it should say "searching for face..." in the top left corner of the video feed. Otherwise, if it's found you, it'll have a frame rate (some number that usually varies between 30 and 130). If all of that seems good then the problem is on the osculator side.

On the osculator side.... is it running? There's a big green pause/start button in the top left corner. If it's running and FaceOSC seems to be working but /found is still 0 then it could be that FaceOSC is sending data to the wrong OSC port or osculator is listening at the wrong incoming port. The default value in the osculator .oscd I provided is 8338. To check the port that FaceOSC is sending to right click on the FaceOSC application and select "show package contents." Then, navigate to /contents/data and open settings.xml. The osc output settings should be set to localhost and 8338.

Let me know if that helps or not...

[Cmac678]

I now have the values changing on the OSCulator. The only thing is that only the fading eyes are showing on the phone. I verified that I am using the right code for face tracking and I am using port 8338 in the code and in faceOSC. What am I doing wrong?

[wkl]

I would try adding some print out statements in the oscEvent function in the processing sketch. At the end of the function (before the last curly bracket) try adding "println" statements for mouthWidth, found, and leftEyebrowPos (do one at a time). You should be seeing values roughly between 0 and 1 in the debug/printout area below the code window in Processing. If you're not, then there's a problem with osculator communicating with processing. In that case make sure that osculator is routing the OSC messages to localhost:8000 (I uploaded a new picture in the last step to clarify this).

It should look like this...
....
} else if (addr.indexOf("/pose/position/0") != -1) {
faceXPos = theOscMessage.get(0).floatValue();
} else if (addr.indexOf("/pose/position/1") != -1) {
faceYPos = theOscMessage.get(0).floatValue();
}
println(mouthWidth);
}

[Emiliano Valencia]

Please help, i cannot find localhost:800, it only sees "will's iPhone"n and "iPod touch" (my ipod)

[wkl]

To add an OSC routing address, click on the message to route and then open up parameters and go to the OSC routing tab.

Under "Will's iPhone" or "so and so's iPod Touch" double click in the empty field. A cursor should appear and you just need to type: localhost:8000. Then ensure that it is selected with the radio button.

[jasechong]

I had that problem. This seemed to work for me:
1) In Osculator, click the Parameters button
2) Change to the OSC Routing tab
3) Check that localhost:8000 is selected, or add it if not in there. I found mine was routing elsewhere and thus Processing and the Mimbo pde script didn't get any events

[Emiliano Valencia]

WOW, i'm defiantly making one, i have an old iPod touch which allready has TouchOSC on it, so i'm using that! 5 stars!

[jimkiller]

I'm having trouble with loading the libraries in processor. I downloaded the oscP5 library, unzipped it and put the folder in sketchbook/libraries but I get an error saying the library is still missing. Any idea what I'm doing wrong? I'm on OSX10.6

[wkl]

Have you restarted Processing? (quit out and then opened it up again).

When you look in your sketchbook do you see the oscP5 library there?
(File->Sketchbook->libraries->oscP5).

If so, open one of the examples and see if that works....

[jimkiller]

I've restarted Processing multiple times. When I look in my sketchbook libraries isn't there but if I look at the same folder in finder then it is there. I've tried opening the examples that come with oscP5 and they won't run either I get the same error...