Instructables

How to make a remote-control sentient web-puppet by hacking Twitter, Google, Skype, Arduino and Processing!

Featured
Picture of How to make a remote-control sentient web-puppet by hacking Twitter, Google, Skype, Arduino and Processing!
How to manipulate a physical object over the web just by using common web services and their accessible data feeds, with some added open source stuff to decode and manipulate that data, and ultimately use the data to move and control physical hardware.

 

Twitr_janus - a prototype web-controlled puppet

This Instructable describes Twitr_janus  - a puppet I made to see if it was possible to mash up free digital web services (Twitter, Google Spreadsheets and forms, Netvibes and Skype) with open source hardware and code (Arduino language and environment, Processing and related Processing libraries) and use them to manipulate an object over the web.

It turned out it was indeed possible!  

See how Twitr_janus works and see how you can use these ideas to build your own remotely-controlled physical objects. It was built from cheap, easily available stuff, some of it salvaged. I made a puppet, because I just like weird, creepy things. The principles it demonstrates could easily be applied to control all sorts of other objects you could build yourself.

Here's Twitr_janus in action, describing itself and how it works...




Summary of what it can do...

A puppeteer can remotely communicate over the web using Twitr_janus' data-activated head.

The puppet can:
  • speak tweets sent to its Twitter account
  • speak longer sentences that have been input as text into a field in a Google spreadsheet
  • move its jaw in time with its speech, using a car door-lock actuator (linear motor) controlled by Arduino which converts audio output into control data to trigger lip-synced movement
  • position its remote-control eyeballs with Arduino-controlled micro servos driven by data from fields in the same Google spreadsheet 
  • be commanded from a control interface hosted in a Netvibes page - created by hosting a hacked version of the standard Google input form (made by modifying the form html to restrict the data values, but riding the Google submit script.
  • be woken up remotely over the web with Skype,  to turn on sight and hearing via an HD video camera
  • use the webcam to allow the puppet operator to see what the puppet eye is looking at can see 
  • use the webcam built-in microphone to allow the puppet operator to hear what the puppet can hear
Note - this instructable is a summary of the major steps that were involved in building a working, data-driven physical object. It introduces the concepts and explains the ways its features are made to work, but does not go into minute detail.

More full detailed descriptions of each step are available in posts on my Making Weird Stuff blog 
There are lots of these - too many for an Instructable. Where relevant though, these detailed discussions are linked to on the steps here. 

A very short summary of the project is also available here:
makingweirdstuff.blogspot.co.uk/2012/11/twitrjanus-overview-november-2012.html

Processing and Aruino code created to make it work is available on GitHub (as straight file downloads). For details see the steps later in this Instructable. Be warned, it's as roughly fashioned as my physical handiwork. Apologies to purist coders. It's freely shared for ideas, but contains some left over functions and snippets that were developed, but not necessarily used. Some were left in the sketches, so copying everything is not recommended. Some of it may be useful. It's built on top of other people's open source stuff so take what you can use.

This project was first shown as to demonstrate a working data-driven object prototype, at the hacking workshop:
"Slack Day" at Museum Computer Network, Seattle 2012.


I'm adding it to Instructables too, as there are loads of people here who might find at least some of it useful. Feel free to hack and modify any ideas here. I learnt a lot doing this from the various open-source communities, especially Arduino and Processing.
 
Remove these adsRemove these ads by Signing Up

Because you have the power over the latex, why not implementing facial expressions with electro-magnets? Use an Arduino Nano (or several clones) to gain space? Get a Raspberry Pi and use openCV for facial recognition? That's what I'm trying to do, thank you for the deodorant-ball trick :-)

Creepy, But interesting...
rosemarybeetle (author)  LucDaRocka11 year ago
Thanks - It is a bit creepy, but that's part of the fun. Puppets are always slightly unsettling.
Hey dude, I like creepy! I work in a haunted house in my town!
rosemarybeetle (author)  LucDaRocka11 year ago
Haunted house? - that's pretty cool. You might like this video I made way back in 1996...
i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now! i must go an kill red tomatoes an green peppers now!
mmmm delicious
Wow! That's really creepy, Reminds me of this deleted scene from a CHILDRENS MOVIE!!! https://www.youtube.com/watch?v=ifeyKhxgHbA
rosemarybeetle (author)  LucDaRocka11 year ago
God, yes. I've seen that. It is incredibly creepy!
BTW, I subscribed to you on Youtube! Have you heard of a show called Courage the Cowardly Dog?
rosemarybeetle (author)  LucDaRocka11 year ago
Hi, I hadn't. Looks very silly, which is always a plus :)
Well it's not made anymore, but you can watch it on youtube... Some of the scenes are very creepy such as this: http://youtu.be/2QU-Kf7ZIYI
rosemarybeetle (author)  LucDaRocka11 year ago
Thanks, yes I looked up a few. I also like PowerPuff girls which has the odd creepy bit, but is mainly just really really silly
Hey, That's old cartoon network for you! Man, I wish I could build amazing things like this, but I don't have the time, tools, or money.
Tomdf1 year ago
That is so incredibly awesome! I want one!
rosemarybeetle (author)  Tomdf1 year ago
Thanks - so is your drinking straw hands thing!
http://www.instructables.com/id/Spring-Loaded-Drinking-Straw-Hinges/
J-Five1 year ago
Cool,
Plus its great for Halloween night
rosemarybeetle (author)  J-Five1 year ago
cheers. it would be very funny to use to freak callers with. nice idea
Like using a cds cell when ever someone breaks the beam it activates it.
rosemarybeetle (author)  J-Five1 year ago
With Arduino, you can use pretty much any component to trigger physical activity.
You could ponder also on:
  • InfraRed/relay detector circuits from outside security lamps (recycle from old ones or buy a cheap one
  • proximity sensors so you can send different escalting messages as people approach - ha ha. You can buy these in component stores online real cheap
Have fun!
Mistwalker1 year ago
That is absolutely terrifying. The most plausible thing I could imagine this horror born of your diseased imaginings saying is "kill me".

Excellent work.
rosemarybeetle (author)  Mistwalker1 year ago
ha ha - praise indeed! halfway through making this Prometheus came out and some parallels there too
Incredibly creepy, and I like it a lot. It gives me fantastic ideas for some sculptures I've been thinking about placing at my driveway gate. By the way, doesn't Janus mean "twin"? Maybe you should make a brother for your creature, or put a second face on him. That wouldn't be creepy at all! Kudos on your work!
rosemarybeetle (author)  KoomoriForge1 year ago
If you want a quick and dirty way to do it, just use the Skype part of it. You can talk directly out of it's mouth from your remote phone/tablet/PC and hear back what they say so you can creep them out by being specific about what they are doing!!
Outstanding! That's exactly what I want. Thanks!
rosemarybeetle (author)  KoomoriForge1 year ago
Great - hope it works out and glad to provide any ideas :)

Another method is to make a translucent face mask and back-project a face onto it that moves - used in attractions like London Dungeon, but quite easy to do. just as freaky!
rosemarybeetle (author)  KoomoriForge1 year ago
Also Janus is the Roman god of doorways and portals, etc. He usually has two faces so he can look in and out. I was considering two heads initially. double-creepy
So why not just give it a RESTful web interface and be done with it? Control it through REST commands and then you can easily interface it with anything on the web through HTTP requests.
rosemarybeetle (author)  deusprogrammer1 year ago
A-ha - I knew a proper developer would pick me up on this.
Thanks - I confess what you're suggesting (or a variation on it) is what I was wanting to do. Ideally a web interface that displays to the puppeteer something useful as a UI (sliders for eye positions, independent text entry for spech, etc etc) and which has some elegant injection via API into the Google sheet was what I had initially fantasised about.

What you're suggesting is no doubt better. All good stuff, but my brain has its paltry edges :)

Any pointers to open source code examples that do the basics of what you're suggesting will be good for others to build on.

Cheers
audhoot1 year ago
That is some cool stuff!
rosemarybeetle (author)  audhoot1 year ago
thanks. All of it is based on free stuff in open source communities, or on services that are fairly easy to piggyback (like Google)
studleylee1 year ago
VeryVeryVery Cool!!!!!!
rosemarybeetle (author)  studleylee1 year ago
thanks :)
There's an amazing quantity of information here. I'm thinking that something like this could be used as a remote presence. Have it on your office desk when you are working from home and you could have meetings with other staff by skype and they could interact with the puppet as if you were actually there?
rosemarybeetle (author)  Dream Dragon1 year ago
Hey, thanks. Yes it took a few months to build, and quite a while to write up!

It was intended to demonstrate some of the possibilities you mention. It is possible to do that, although it would need more work on the input interface. However, you could use direct Skype for talking, but Google for positioning (with a bit of work on making an api call directly into the spreadsheet with a real developer working on it, rather than my tinkering)
Kiteman1 year ago
That's an excellent project, but, dang, that's one scary puppet!
rosemarybeetle (author)  Kiteman1 year ago
Hey thanks.

Lots of fun.
I quite like ceepy things - It can see and hear as well as speak and look at you - ha ha ha.

You might also like my reading lamp:
http://www.instructables.com/id/Arduino-controlled-animatronic-wooden-head-readin/