I'll help you pick out a good toy to rescue, and then guide you through the process I used to successfully hack all of the buttons and switches to make something really cool and useful. We'll rip out the old, useless guts of the toy and replace it with a cheap microcontroller that is capable of sending and receiving MIDI messages to a PC, which will do the actual sound synthesis for us. I'll discuss the ins and outs of how to do this using code, and hopefully help you improve your project-making skills along the way!
Example of finished MIDI controller
What is a MIDI controller?
The Musical Instrument Digital Interface (MIDI) protocol was developed by professional audio technicians as a way for their various electronic devices (drum machines, synthesizers, samplers and more) to communicate by sending messages back and forth. Messages like, “play this musical note” and “we’re now in the 3rd beat of the 2nd measure” are transmitted between devices to help keep all of the devices in sync and reduce the amount of work that people need to do.
Every musical device out there today can be classified as a specific type of device, such as a sequencer, synthesizer, sampler, controller or others. In this article, we will be converting a toy into a MIDI controller. A MIDI controller typically has no ability to produce any noise or manage musical notes into a piece of a music. Instead, it’s more like a keyboard or a mouse; they tell a more intelligent device about changes in it’s own systems (i.e. “user pressed the A button”).
You can learn a great deal more about the MIDI using a variety of resources. Since MIDI has been around for quite some time, there is a great deal of information available on the internet, such as:
For personal reference, I ordered the following book and have been very happy with it: http://www.amazon.com/MIDI-Power-Comprehensive-Robert-Guerin/dp/1598630849/
Step 1: Acquiring a Suitable Toy
What to look for
- Relatively low number of buttons. Less than a dozen is a good idea.
- Very few or no sliders or knobs. You can utilize these in a MIDI controller, I just won't cover it here
- A big enough enclosure to accommodate a few extra wires and a Teensy microcontroller
Where to find a good toy
Don’t over-think this process too much; if its a toy that has electronics inside of it, you can convert it into a MIDI controller! If you’re flipping through pages of Google results or Amazon products trying to find the toy that is “just right”, stop it. The best place to get fantastic toys from is your local Goodwill, Salvation Army or other local thrift store, and are they only places I like to go to when looking for toys. Buying a brand new retail toy from Wal-Mart will only contribute to wasteful consumerism and put a hole in your wallet.
The toy guitar that I will be using for demonstration in this article was purchased from my local Salvation Army for $1.95! Thousands upon thousands of cheap toys are thrown out all the time by families all over the country, usually for trivial reasons like cracks or a broken button. Believe it or not, the most common cause for toys being thrown out is simply that the batteries are dead!
Introducing my toy guitar
I stumbled across this beauty one day at my local Salvation Army, and knew I had to have it as soon as I found it. In fact, I decided to do this article after I found this guitar, because I thought it would make such a cool MIDI instrument! The guitar features a large variety of pieces that can be pressed, spun, rocked and tilted. It even includes some neat lights and unusual features (like that gray wheel at the bottom – it spins!). When I brought it home, I quickly found out why it was discarded; there was a crack in the plastic of the fretboard, making the whole thing flex in odd ways. Furthermore, the batteries were dead. I applied super-glue to crack and removed the batteries, and the guitar was good as new!
Step 2: Learning About How Your Toy Works
Buttons made from PCBs and silicone domes
In many bulk-manufactured devices today, buttons look nothing like the buttons you can buy from RadioShack. Instead, interlocking (but disconnected) traces are etched into custom-made printed circuit boards (PCBs) to create a normally-open (NO) type switch. An example of these interlocking patterns is shown in the fifth picture of this step. A silicone dome, housing a small piece of conductive material, is then fitted onto the PCB in such a way that when the dome is pushed down, the conductive material bridges the various interlocking traces on the PCB, thus completing the circuit and closing the switch.
By adjusting the height, size, materials and other properties of the silicone dome, manufacturers can alter the 'feel' of a button with ease. You are likely to find these all over your board, because they are quite cheap and useful.
Leaf spring switches
Another common kind of switch you might find, which provides a different kind of 'feel' and I imagine is a little more durable, is a leaf spring switch, as seen in the seventh picture of this step. If you have a strip of metal, it's thickness and material characteristics will determine how easy it is to bend. The right combination of this properties will result in a strip that feels somewhat 'springy', meaning it will bend a little bit, but will keep bouncing back to it's original form when released. If you place a small piece of metal at the point where the strip bends to, you can create a switch that springs open until you close it.
Understanding the web of traces
Each board should have some wires coming from it, which carry electrical signals to and from the board and other parts of your toy. Generally, you can easily turn the circuit board over and study the traces to figure out which wire goes to which component. It may help you to draw a picture of the circuit board, then make notes about which wires go where, and trace out the path of each signal around the board. Remove the silicone domes temporarily to make sure that you aren't missing any 'ninja' traces underneath them.
Take some time to make diagrams of each of your circuit boards and determine which of the board's wires are for ground and which are for signals, as well as what these signals represent. Generally, you should be able to identify at least one wire that is shared by several components - this is the ground wire. You should also see at least one individual signal wire per component.
We will take a look at what to do with these wires in Step 4, so keep those diagrams nearby!
Highly-specialized sound-making circuitry
In addition to the buttons and switches throughout your toy, you should also have a board which contains a small, round black dot. This is a highly-specialized chip that contains all of the sound samples and logic required to turn the signals from the various buttons into actual noises. Furthermore, you will be able to find a speaker of some sort somewhere in your toy. In the next step, we'll remove these components, so that we can put our own microcontroller in it's place.
Step 3: Removing the Old Circuitry
Another key identifying characteristic of the main sound-generation board is a small, oddly shaped black chip. It will look like someone dripped a small amount of black candle wax onto the board and let it form a small circular glob. This component will likely have many small copper traces coming out of it. This little chip actually contains all of the logic to interpret all of the various buttons, sliders and other components throughout the toy and make noise using the speaker. Talk about no-nonsense manufacturing!
So what can you do with this board? Well, not much. You may have some success exploring circuit bending techniques, but for the purposes of this Instructable, it's useless.
Cut each of the wires coming from this circuit board and remove it. Also take care to remove the speaker, as we won't be using it either.
Credit to David Storch for the photo, from https://wiki.brown.edu/confluence/display/mcm0750/Boot+with+Knob+and+Buttons
Step 4: Understanding Switches and Pull-ups
If you connect a normally-open switch to a digital device like a microcontroller, then attempt to read the state of that switch, you will actually receive a somewhat random value that you cannot predict. A digital device must be connected to either ground (to read "0") or some voltage like +5V (to read "1"), so if it is not connected to either, it cannot be sure what its connected to. This phenomenon is called floating, and is very important in the world of microcontroller circuits. We can solve this problem using a single resistor, either tied to +5V (pull-up) or ground (pull-down). Since we're already connecting one end of the switch to ground, we need to use a pull-up resistor.
What is a pull-up?
Take a look at the picture for this step, and you'll see a schematic of what a pull-up resistor and a switch look like together. The switch, SW1, is a button in your toy, which we are connecting to ground. The resistor, R1, links the other side of the switch to +5V, so the microcontroller will only ever see one or the other. However, using modern microcontrollers, we don't have to add a resistor to every single button in the toy - the microcontroller already has one built into each of it's pins for this exact purpose! Maybe I'm easily amused, but that's pretty cool.
Step 5: Mapping Out All the Parts
Identifying the components you want to use.
Begin by identifying everything that you want to use. I have found that the technique that works best for me is to pick up a piece of paper and draw a very simplified picture of what I see inside the toy, and try to identify which things I’d like to use, and which things I don’t want to use. These sketches are attached to this post, so check them out as you read this step.
Notice that at the bottom of each page, I am keeping track of how many inputs and outputs I have identified in my drawings. This will help you determine what microcontroller to buy later!
An important bit of information that should help you in identifying how your circuit boards work is to remember that every single board will have at least one ground wire and at least one signal wire for each component.
Special note about charlieplexing
For circuit boards that have many buttons on them, you may run into a technique that can be quite confusing for beginners called charlieplexing. Essentially, it’s a technique whereby several buttons share the same signal and/or ground lines. In this case, we must identify more than one ground line and connect one wire to each. Using a microcontroller, we can enable or disable these ground lines as we please, and thus figure out which button is sending information along a signal line. If you have never encountered this technique before, and find a circuit board that you think uses it, you can choose to either ignore that circuit board (and thus it’s buttons), or pick up another cheap toy.
Step 6: Wiring Up a Common Ground
Note about multiple grounds and charlieplexing
If you have one or more circuit boards that use multiple grounds (charlieplexing) and you DO want to use them, do not connect them together, but rather consider them signal wires, and thus leave them alone for now.
Step 7: Attach Wires to Each of the Components (optional)
Since these wires are quite frustrating to solder, strip and manipulate, you may decide to completely replace the wires with the hookup wire of your choice. Personally, I love the solid core wires that ethernet cable is made out of - once they are debraided and straightened out, they make for excellent hookup wires, and are quite cheap!
In the photos, you'll notice that I also labeled each of the wires and groups of wires using masking tape - it helps me keep track of which wire is which (since I have so many), and keeps me organized between hacking sessions.
Step 8: Testing Out Each Component
This will help you identify simple problems with either your wiring, soldering or mapping without having to worry about whether or not the Arduino code or MIDI functionality is to blame.
Use the Arduino code attached to this sketch to test out each wire from the previous step individually.
1. Upload the sketch to your Arduino
2. Connect the toy's common ground to your Arduino's ground
3. Connect the Arduino pin coded into the sketch to a signal wire of a component
3. Open the Serial Monitor
4. Press the button on your toy and make sure the stream of numbers changes from 1 to 0
Help! One or more buttons do nothing!
First, double-check your wiring. Do the solder joints look good? Are the bare parts of any wires touching? Do any wires have any cuts or nicks in them? Are both ends of the wire firmly connected to their parts? Is your common ground connected to your Arduino?
If everything checks out but the wire still doesn't work, try moving on to another wire, just to make sure that the code and PC connection are correct. If you can get at least one other wire working, you can be sure that your Arduino environment and USB connection are OK. If you absolutely cannot get it to work, you may have to just disconnect it and move on - sometimes fixing low-grade consumer toys is more trouble than its worth. Feel free to leave a comment on this step with any more tips or troubleshooting hints!
Step 9: Prepare to Add a Microcontroller
Choosing and acquiring your microcontroller
The main Arduino board that you can buy is wonderful for beginners to learn basic concepts about electronics and to prototype their ideas without much hassle. However, it can be quite a challenge to fit an entire Arduino board into your toy! Furthermore, it is presently not as easy as it should be to make your Arduino board act like a genuine MIDI device.
For my project, I chose to use the Arduino-compatible Teensy++ 2.0 from PJRC, and I am going to highly recommend that you use one as well. In addition to being extremely small, the Teensy series boasts the ability to emulate different types of USB devices, like keyboards, mice and (you guessed it) MIDI devices! In my opinion, these factors make Teensy the perfect choice for this kind of project, so go and get yourself one!
Setting up your Arduino environment to work with Teensy
To work with your Teensy in the Arduino IDE, you need to install a special add-on called Teensyduino. This adds a few new menu items to your "Tools" menu, and adds some really useful functionality to your environment. Read the Teensyduino page for installation instructions, and check out the various sections of the menu on the left to learn about all the cool things you can now do.
Planning out how your wires will connect to your microcontroller
Regardless of how complicated your toy is, it would be a good idea to make yet another diagram documenting which wire in your toy should connect to each pin of your microcontroller. PJRC supplies some really helpful diagrams showing all of the pins of the Teensy microcontrollers and their labels in Arduino. Print one out, or use it as reference to draw your own picture, and label the pins according to how you want to connect your wires.
My toy had a lot of inputs and outputs, so I wanted to keep very careful notes about which wire goes to what. Therefore, I created a simple spreadsheet to keep track of every wire, their labels, functions, Teensy connection and MIDI note value. You may not need to go to these lengths if your toy is relatively simply, but I wanted to share my process just in case.
Step 10: Install the Teensy and USB Cable
The most important thing about adding a USB cable to your toy is to be very careful about the mechanical stresses it might come under. Always imagine what might happen in a worst-case scenario; maybe you step on the cable while you’re moving around, or it gets caught on something and yanks very hard on your toy. For this reason, I recommend securing placing the Teensy where you want it in the toy, then securing the USB cable to the case to provide enough slack to reach the Teensy. The more you can secure the USB cable to your toy’s enclosure, the less stress it will put on your Teensy. It’s much better to have the USB cable simply become unplugged than for the socket of your Teensy to snap off, or the board to crack!
For now, don't worry about affixing the Teensy to the case - you will be turning it over and moving it around as you solder wires to it in the next step.
Step 11: Wiring Up the Microcontroller
Start small, test often.
Pick a wire, or group of wires, and find a good way to route it that is not going to make it hard to close the toy back up again. Find empty spaces that don’t have anything in them, and try to avoid posts that are used for screws. Your two best friends in this process will be sticky tack and masking tape. Take a look at my process photos below to get an idea of how I went about laying out my wires and soldering them to the microcontroller.
NOTE: I recommend testing each pin as you solder it using the procedure in the next step!
Step 12: Write Code to Test Each Pin
Download the Arduino sketch attached to this step and open it up. You'll notice it looks a bit more complicated than the code from Step 7, but it is for good reason. Firstly, the code includes a debouncing routine that is capable of checking the state of an arbitrary number of pins. Secondly, the code uses an array to hold all of the pin numbers, so that you only have to edit one variable whenever you want to add or remove pins from your toy, rather than diving into the entire sketch!
Debouncing may be necessary with certain kinds of buttons because of something called mechanical contact chatter, or “bouncing”. This occurs when the two metal pieces of a switch make contact, and then “bounce” off each other for a brief moment. This happens so quickly that you won’t notice it in the Serial Monitor or by touch, you would only really know about it from the problems it would cause, or by looking at the signal on an oscilloscope.
There are many techniques one can use to debouncing inputs, but I am using a technique outlined by Limor "ladyada" Fried here: http://www.adafruit.com/blog/2009/10/20/example-code-for-multi-button-checker-with-debouncing/
Using the sketch
I recommend testing one pin at a time, just to make troubleshooting easier. Update the buttons array to contain the pin number of the pin you want to test, then upload the sketch to your Teensy. Open the Serial Monitor and watch for messages sent from your Teensy. The sketch will send a message every time a button is pressed and released, indicating the pin number that recorded the event. Once you know a pin works, you can keep it in the array. When you move on to the next pin, add it to the end of the array, so that by the end, all of your buttons are added to the code.
Step 13: Securing the Teensy Inside the Case
To secure my Teensy, I simply drilled small holes through my case on either side of the board (leaving a little room for wiring), then slipped two zip ties around the Teensy. Be careful not to excessively tighten the zip ties, they just need to be tight enough to hold the Teensy in place and keep it from flying inside your toy.
Step 14: Adding MIDI Functionality
MIDI at a glance
MIDI can be a complex and immensely powerful protocol, and as such, there are many kinds of messages you can send from your Teensy. If you are so inclined, and want take your MIDI toy to the next level, I suggest first learning more about what you can send from your Teensy from this helpful page from PJRC about using MIDI on the Teensy.
To keep things simple, however, we are only going to focus on two specific types of messages in this Instructable: note on and note off. There’s no real mystery to these messages, they do exactly what you expect them to; they tell the device your controller connected to that it should either start or stop playing a specific musical note. You can learn about other types of messages, like control changes (CC) and sysex messages to enable more advanced effects, but we'll just keep it simple for now.
Using the Arduino sketch
The Arduino sketch attached to this step should look familiar, as it is based heavily on the sketch used in Step 11. All of the debouncing code is still in place, but a couple new variables have been added:
midiNotes array = array of MIDI note numbers that should be associated with the pins listed in the buttons array. For example, if the code sees that you are pressing or releasing buttons, it will send appropriate MIDI messages to turn on and off the MIDI note in midiNotes.
channel = all MIDI messages must been sent on a certain channel. This allows more than one device to be plugged into a MIDI chain and interact with sequencers and synthesizers.
Don't forget to change the USB Type of your Teensy!!
Your Teensy is capable of sending MIDI messages only when it is configured to be a MIDI device! To do this, just go to the Tools > USB Type menu and select "MIDI". The downside to this is that while in MIDI mode, you cannot send or receive Serial messages. This is why we did all the debugging in previous steps - you should already figured out if all of your pins work, we're just adding to the working system in this step!
Step 15: Testing for MIDI Messages on Your Computer
For the rest of us, however, open-source software can help us out a little bit. As I mentioned, there are many ways to look for MIDI messages, including writing Processing code, using a small MIDI utility like MIDI-OX or using popular synthesis programs like PureData or VVVV. My personal favorite environment is Processing, but that can take a bit of work to just see if MIDI messages are coming through. Therefore, we'll use PureData to get up and running quickly.
Using Puredata to check for MIDI messages
First, head on over to the PureData website and download a copy of the latest version of PD for your operating system. Once you've installed it and opened it up, you can easily check for MIDI messages using a built-in function of the IDE.
Go to Media > MIDI Settings and select the Teensy from the menu (for me, it showed up as "OSS MIDI Device #1")
Next, go to Media > Test Audio and MIDI and watch the boxes in the lower-left part of the window that pops up, labeled MIDI In. Each time you press and release a button on your toy, you should see some changes occur. Numbers should change inside of the two boxes directly under the "stripnote" box (corresponding to MIDI note number and velocity), and the circle at the bottom should turn black and white from time to time (to indicate note on and note off).
Step 16: Using Your New MIDI Controller
To create music with your MIDI controller, you need to interface it with a synthesizer of some sort which can translate MIDI messages into actual noises. There are many, many great pieces of software to help you with this, but they can get quite costly. Here is a list of programs worth checking out that can help:
- Ableton Live!
- GarageBand (Mac, free)
- Puredata (free)
Since MIDI messages can be read by nearly any software framework capable of serial communication, you can actually use MIDI for much more than music! You can use it in any situation where you would like to control some sort of activity through a physical interface, such as the following:
- Lighting and home automation (like http://www.midi.org/aboutmidi/halloween.php)
- Controlling video games. Re-program the Teensy to use the Keyboard and Mouse objects to send keystrokes like WASD, space and mouse clicks. Ever wanted to play Skyrim on a guitar?
What I've used my controller for
From the get-go, I was intending to use my MIDI controller to make sounds and, hopefully, music. However, I don't have much of a background in this area, so I'm more just messing around at this point. In the video below, I've connected my MIDI controller to Pure Data and created a simple synth patch based on this helpful article from Anthony Mattox.
Sometime in the next couple weeks, I hope to give the controller to a friend who dabbles in professional audio production and let him make something cool with it, but here is what I have for now:
Step 17: Future Improvements
Pitch bends and after-touch
In MIDI, you can also send messages that alter the properties of notes after they have been sent. Pitch bend messages alter the pitch of a note, while maintaining the same velocity. After-touch messages alter the velocity, and leaves the pitch the same. You can use these messages to create cool kinds of continuous controllers, like violins and cellos.
Control changes (CC) and System Exclusive (SysEx) messages
You can also send a variety of very specific messages to tell your synth what kind of behavior you want your controller to have. You can use these messages to mimic modulation wheels, sustain pedals and more.
Receiving MIDI messages on a Teensy
In addition to sending MIDI messages, the Teensy can also receive MIDI messages, which can be really cool if you have LEDs, motors or other actuators that you want to control from your computer. All you really have to do is implement a method corresponding to the message you want to receive, then use the parameters that are passed into it however you want to control the IO of the Teensy! Learn more here: http://www.pjrc.com/teensy/td_midi.html