How to make a remote-control sentient web-puppet by hacking Twitter, Google, Skype, Arduino and Processing!

How to manipulate a physical object over the web just by using common web services and their accessible data feeds, with some added open source stuff to decode and manipulate that data, and ultimately use the data to move and control physical hardware.

 

Twitr_janus - a prototype web-controlled puppet

This Instructable describes Twitr_janus  - a puppet I made to see if it was possible to mash up free digital web services (Twitter, Google Spreadsheets and forms, Netvibes and Skype) with open source hardware and code (Arduino language and environment, Processing and related Processing libraries) and use them to manipulate an object over the web.

It turned out it was indeed possible!  

See how Twitr_janus works and see how you can use these ideas to build your own remotely-controlled physical objects. It was built from cheap, easily available stuff, some of it salvaged. I made a puppet, because I just like weird, creepy things. The principles it demonstrates could easily be applied to control all sorts of other objects you could build yourself.

Here's Twitr_janus in action, describing itself and how it works...

Summary of what it can do...

A puppeteer can remotely communicate over the web using Twitr_janus' data-activated head.

The puppet can:

• speak tweets sent to its Twitter account
• speak longer sentences that have been input as text into a field in a Google spreadsheet
• move its jaw in time with its speech, using a car door-lock actuator (linear motor) controlled by Arduino which converts audio output into control data to trigger lip-synced movement
• position its remote-control eyeballs with Arduino-controlled micro servos driven by data from fields in the same Google spreadsheet 
• be commanded from a control interface hosted in a Netvibes page - created by hosting a hacked version of the standard Google input form (made by modifying the form html to restrict the data values, but riding the Google submit script.
• be woken up remotely over the web with Skype,  to turn on sight and hearing via an HD video camera
• use the webcam to allow the puppet operator to see what the puppet eye is looking at can see 
• use the webcam built-in microphone to allow the puppet operator to hear what the puppet can hear

Note - this instructable is a summary of the major steps that were involved in building a working, data-driven physical object. It introduces the concepts and explains the ways its features are made to work, but does not go into minute detail.

More full detailed descriptions of each step are available in posts on my Making Weird Stuff blog 
There are lots of these - too many for an Instructable. Where relevant though, these detailed discussions are linked to on the steps here. 

A very short summary of the project is also available here:
makingweirdstuff.blogspot.co.uk/2012/11/twitrjanus-overview-november-2012.html

Processing and Aruino code created to make it work is available on GitHub (as straight file downloads). For details see the steps later in this Instructable. Be warned, it's as roughly fashioned as my physical handiwork. Apologies to purist coders. It's freely shared for ideas, but contains some left over functions and snippets that were developed, but not necessarily used. Some were left in the sketches, so copying everything is not recommended. Some of it may be useful. It's built on top of other people's open source stuff so take what you can use.

This project was first shown as to demonstrate a working data-driven object prototype, at the hacking workshop:
"Slack Day" at Museum Computer Network, Seattle 2012.

I'm adding it to Instructables too, as there are loads of people here who might find at least some of it useful. Feel free to hack and modify any ideas here. I learnt a lot doing this from the various open-source communities, especially Arduino and Processing.

Step 1: What you can get from this - the basic elements that make Twitr_janus work

Twitr_janus was an attempt to see if I could hack up web stuff to make a remote control puppet - a sort of physical avatar. It is really just a puppet head, a bit like a ventriloquist dummy, but one that is wired up to the web to make it come alive!

As a bit of hackery, it worked a treat.

In this Instructable I have separated out the various elements that make it work, so you can steal any ideas that help. I have tried to keep it simple here. If anything is of potential use, the more detailed explanations are usually available in posts on my Making Weird Stuff blog. Links are provided here where relevant.

The basic building blocks that make Twitr_janus work are: 

• The physical head, built from papier mache, and moulded plastic (hot glue in fact)
• Input sensors for sight and hearing (using a common webcam with built-in mike)
• servos for controlling eyball and jaw movement
• LEDs for indicating when web data was being received
• A connected computer running a programme for
  • listening to the web for command data to control the head (using url parsing in a Processing sketch) by communicating over a serial port with an arduino board
  • allowing text to be spoken (using a text-to-speech library in the Processing sketch)
  • running Skype to allow the head to relay what it can see and hear back to the operator
  • sending audio signals to the arduino
• An arduino board listening for control data from the mother computer:
  • to convert it into output control data for the eyeball servos
  • to convert analogue audio signals into control signals for the jaw motor
  • to activate separate LEDs to indicate the sources of data (Google RSS and Twitter API)
• Remote operation using
  • any web connected device capable of running Skype, Twitter and Google Doc. This was possible with an iPad or even iPhone, but was only practical with a full size computer (laptop)
  • a control form embedded in a Netvibes html page, pimped to piggy-back the back of a Google Spreadsheet form submit script

This instructable shows how, in case you ever might be inclined to nick some of the ideas.
The most useful stuff includes:

• how to strip data from RSS feeds and APIs using Processing to repeatedly listen to the web
• how to use expressions in Google Spreadsheets to pass special delimited strings to differentiate different data 
• how to set up a custom html form with pre-set control values, that uses a Google spreadsheet form submit script to   the spreadsheet
• how to detect new data, but ignore old messages that have been received already
• how to send the data over a serial port to an Arduino
• basic servo, relay and LED control with the Arduino
• text-to-speech conversion using free Processing library
• audio peak detection with Arduino to trigger a servo to operate the jaw
• how to remote kick-start a dormant Skype contact to wake up sight and hearing
• making a lightweight skull using papier mache over a remeable model
• using hot-glue as a casting material to recreate a face mask model
• lots of other silly minor details that make it all work

Explanations in this Instructable cover most things briefly. For some components, the full details on the development of specific elements may be via links to posts on my Making Weird Stuff blog. These offer much more detailed explanations.

Twitr_janus is a reflection on how we take for granted "free" web communication tools - handy services like Twitter, Google, Skype - and how when using them, we develop online personas. However these personas that represent our words are usually no more than text boxes on screen with an associated and usually rather dull 2D photo.

Twitr_janus was an attempt to make a real physical representation of online activity. The challenge was to see if I could make such a monstrosity work using the common, but actually quite advanced, data-communication channels and features that web-based services provide for free.

Comments

[lukev2]

Creepy, But interesting...

[rosemarybeetle]

Thanks - It is a bit creepy, but that's part of the fun. Puppets are always slightly unsettling.

[lukev2]

Hey dude, I like creepy! I work in a haunted house in my town!

[rosemarybeetle]

Haunted house? - that's pretty cool. You might like this video I made way back in 1996...

[lukev2]

Wow! That's really creepy, Reminds me of this deleted scene from a CHILDRENS MOVIE!!! https://www.youtube.com/watch?v=ifeyKhxgHbA

[rosemarybeetle]

God, yes. I've seen that. It is incredibly creepy!

[lukev2]

BTW, I subscribed to you on Youtube! Have you heard of a show called Courage the Cowardly Dog?

[rosemarybeetle]

Hi, I hadn't. Looks very silly, which is always a plus :)

[lukev2]

Well it's not made anymore, but you can watch it on youtube... Some of the scenes are very creepy such as this: http://youtu.be/2QU-Kf7ZIYI

[rosemarybeetle]

Thanks, yes I looked up a few. I also like PowerPuff girls which has the odd creepy bit, but is mainly just really really silly

[lukev2]

Hey, That's old cartoon network for you! Man, I wish I could build amazing things like this, but I don't have the time, tools, or money.

[Tomdf]

That is so incredibly awesome! I want one!

[rosemarybeetle]

Thanks - so is your drinking straw hands thing!
http://www.instructables.com/id/Spring-Loaded-Drinking-Straw-Hinges/

[J-Five]

Cool,
Plus its great for Halloween night

[rosemarybeetle]

cheers. it would be very funny to use to freak callers with. nice idea

[J-Five]

Like using a cds cell when ever someone breaks the beam it activates it.

[rosemarybeetle]

With Arduino, you can use pretty much any component to trigger physical activity.
You could ponder also on:

• InfraRed/relay detector circuits from outside security lamps (recycle from old ones or buy a cheap one
• proximity sensors so you can send different escalting messages as people approach - ha ha. You can buy these in component stores online real cheap

Have fun!

[Mistwalker]

That is absolutely terrifying. The most plausible thing I could imagine this horror born of your diseased imaginings saying is "kill me".

Excellent work.

[rosemarybeetle]

ha ha - praise indeed! halfway through making this Prometheus came out and some parallels there too

[deusprogrammer]

So why not just give it a RESTful web interface and be done with it? Control it through REST commands and then you can easily interface it with anything on the web through HTTP requests.

[rosemarybeetle]

A-ha - I knew a proper developer would pick me up on this.
Thanks - I confess what you're suggesting (or a variation on it) is what I was wanting to do. Ideally a web interface that displays to the puppeteer something useful as a UI (sliders for eye positions, independent text entry for spech, etc etc) and which has some elegant injection via API into the Google sheet was what I had initially fantasised about.

What you're suggesting is no doubt better. All good stuff, but my brain has its paltry edges :)

Any pointers to open source code examples that do the basics of what you're suggesting will be good for others to build on.

Cheers

[KoomoriForge]

Incredibly creepy, and I like it a lot. It gives me fantastic ideas for some sculptures I've been thinking about placing at my driveway gate. By the way, doesn't Janus mean "twin"? Maybe you should make a brother for your creature, or put a second face on him. That wouldn't be creepy at all! Kudos on your work!

[rosemarybeetle]

Also Janus is the Roman god of doorways and portals, etc. He usually has two faces so he can look in and out. I was considering two heads initially. double-creepy

[rosemarybeetle]

If you want a quick and dirty way to do it, just use the Skype part of it. You can talk directly out of it's mouth from your remote phone/tablet/PC and hear back what they say so you can creep them out by being specific about what they are doing!!

[KoomoriForge]

Outstanding! That's exactly what I want. Thanks!

[rosemarybeetle]

Great - hope it works out and glad to provide any ideas :)

Another method is to make a translucent face mask and back-project a face onto it that moves - used in attractions like London Dungeon, but quite easy to do. just as freaky!

[audhoot]

That is some cool stuff!

[rosemarybeetle]

thanks. All of it is based on free stuff in open source communities, or on services that are fairly easy to piggyback (like Google)

[studleylee]

VeryVeryVery Cool!!!!!!

[rosemarybeetle]

thanks :)

[Dream Dragon]

There's an amazing quantity of information here. I'm thinking that something like this could be used as a remote presence. Have it on your office desk when you are working from home and you could have meetings with other staff by skype and they could interact with the puppet as if you were actually there?

[rosemarybeetle]

Hey, thanks. Yes it took a few months to build, and quite a while to write up!

It was intended to demonstrate some of the possibilities you mention. It is possible to do that, although it would need more work on the input interface. However, you could use direct Skype for talking, but Google for positioning (with a bit of work on making an api call directly into the spreadsheet with a real developer working on it, rather than my tinkering)

[Kiteman]

That's an excellent project, but, dang, that's one scary puppet!

[rosemarybeetle]

Hey thanks.

Lots of fun.
I quite like ceepy things - It can see and hear as well as speak and look at you - ha ha ha.

You might also like my reading lamp:
http://www.instructables.com/id/Arduino-controlled-animatronic-wooden-head-readin/