# Earfingers: Hear With Your Hands

6,471

47

15

## Introduction: Earfingers: Hear With Your Hands

First and foremost, I must acknowledge that I am standing on the shoulders of giants, and that every giant is standing on the shoulders of giants (such as all contributors to instructables). If it weren't for the unknowably many people who had the mindfulness to freely share information, this would have been utterly impossible; I reckon the same if even a handful of these people chose otherwise. So, if you like this, don't just thank me, thank a history of humans and the tremendous power of the freedom of information.

Are you bored of sensing things in the same way they've been sensed for the history of humanity? Have the known forms of interacting with computers lost their flair? Is an insufficient auditory neurosystem bogging you down? Then I think I have the project for you. Why not give your ears a rest, and let your fingers do the listening for a change!

Let's prepare ourselves with some background information. Human hearing relies on a bunch of sensors in a structure called the cochlea, which, by way of a very remarkable structure, converts variations of air pressure into neural impulses. In mathematical terms, the cochlea decomposes a waveform into a finite number of dimensions, about 3600 per ear. Have you ever wondered what the sampling frequency of the ear is? It's not a very well-formed question, so to give a not very well-formed answer: from 50 to 250 Hz. Another interesting fact is that the high limit of reasonably good hearing is 20,000 Hz. Together, this means that a sensor at maximum speed is only detecting 1 out of every 80 wavelengths. This is peculiar. It would be like looking at a screen and only being able to see 1 circle every second when 80 circles are shown, and being able to tell anyway that there are actually 80 circles. How can we experience something faster than that which defines our experience? And how can we experience it as continuous when the signal keeps cutting out? With mathemagic. The cochlea's very special coiled construction is a physical, mechanical implementation of a process called the wavelet transformation, which really might as well be magic; conceptually, it's like using part of a torn picture to reconstruct the lost part.

Another key piece of this project is the tactile sensory system. It is an interesting fact that touch sensors also operate in the range of 50-250 Hz. Actually, throughout the body neurons behave in the exact same way. You can move a whole chunk of brain to a completely different spot and have it function as the piece it replaced--scientists have actually done this! As such, it is reasonable to suspect that touch receptors can convey the same information as the cochlea and in the same way, just to different places. Furthermore, some research has indicated that touch does activate some of the auditory processing part of the brain, and one of the best known properties of the brain is that it is extraordinarily adaptable. Thus, we have reason to suspect that you might actually be able to hear through touch! That is, to not just feel the vibration of sounds, like touching a speaker, but to feel sound .

With the idea in place, the rest is simple, right? All we need to do is build a vibrotactile human/computer interface and write some code to send wavelets of sound to the fingertips. The brain will never see it coming! Clearly not, because we're not working with sight (yet); the real question is, will the brain feel it coming, or will the brain hear it coming?

## Step 1: Building Tactors

Plenty of research has been done into tactile interfaces. Because of this, there is a fancy scientific name for the critical components here: tactors, which I suspect is a portmanteau of tactile and motor. You could also call them solenoids, linear actuators, moving magnet voice coils, vibrotactile transducers, etc. The specific requirements of their construction requires that they be built by hand, exciting!

You will need:
tiny powerful cylindrical magnets, I used 1/8" diameter with 1/8" height to keep the scale workable, and added a 1/8"x1/16" for more height/mass/magnetic force and to mark the pole
thin walled non-ferrous tubing with an inner diameter that matches the outer diameter of the magnets (ie K&S stock #103, can be found at hardware or hobby/craft stores)
plastic washers that can be press fit onto the tubing
dowel that can be put into the tubing (paper, cut down toothpicks, erasers, etc could also work)
a flexible membrane (latex glove, balloon, etc.)
small rubber bands or heatshrink tubing
electrical tape
some motor winding wire, I used 34 gauge
a lighter and sandpaper
a long bolt with nut to pass through the cut down tubing
a few washers to pass over the tubing
an electric drill with continuously variable speed (most of them)

We are building a bunch of small solenoids that will push the magnets against the skin with just the right amount of force. This will require building a spool, winding that spool, inserting the magnets, and covering the assembly so the magnet can't escape. Note that you can get away with a fair degree of ingenuity here; I made quite a few prototypes leading up to this using things I had laying about.

1. Cut the tubing: Ideally this is done with a tube cutter, because the tube needs to stay round enough that the magnet can move through it. I made the tubes a bit longer than necessary, 3/4", to make them easy to mount later on, just by drilling holes in some wood and pressing in the spare end. Round and ream the tube as needed, a good way to do this is with a drill bit that matches the inner diameter of the tube.

2. Press on the plastic washers: Unless you have an amazing supply at hand, you will probably have to find some with a smaller inner diameter and drill those out. The only way I know how to do this is with some pliers and a hand drill--please, find a better way (and let me know!) or be very careful. We need a strong fit because these will support the coil, and later on, fingers. The outer diameter of the washers and the space between them is going to determine how much wire you can wind and how much power will be needed. You want to keep the power requirements low but still be able to move the magnet well enough; I determined this experimentally, and came up with between 2.5 and 3 spans across a half an inch.

3. Conjure a winding clamp. Use washers to fill the extra tube and to grip the end of the winding wire. Clamp the bobbin with the nut and bolt, there needs to be enough force to stop the tube from rotating while getting a good, tight coil. Be sure to leave enough spare to reach the terminating points later.

4. Wind the winding wire. Clamp the tail or nut of the bolt in the drill and start winding. The wire must be tight and even to achieve the optimum magnetic field. This is probably the most challenging part of the build, so be patient and take your time. Each solenoid should have about the same number of windings in order to make them all produce a similar force; I did this by placing the plastic washers the same distance apart, then spanning that distance the same number of times (2.5 to 3) since that's much easier to keep track of. You can mark the leads and wrap according to the right hand rule, or just change the polarities around later on. This would be a good time to test the continuity and resistance of the solenoid; most of my solenoids came out with a resistance of 2 ohms, which was from around 2.5 spans.

5. When you're done winding, you'll want to secure all that hard work from coming uncoiled. I like to hold it with a little bit of electrical tape first, then put heatshrink over that. Be completely certain that you've removed the insulation from the ends of the wire by burning it off and then taking off what remains with sandpaper.

6. Finishing touches. Put a 1/2" long piece of dowel in the tube so that the magnet rests above the halfway point of the coil and so that it can only go one way, then add the membrane to cover the top so the magnet can only go so far in that direction. Latex gloves will work, but they tend to break down quickly, so a cut up balloon will probably do better. This can be held on with anything non-ferrous , solid aluminum or copper wire works well. Typically the magnet would be secured to the membrane, but there was no obvious way of doing that on this scale. You can test them by rubbing the leads on a battery; these are fundamentally similar to small speakers, so with the right polarity you will hear the scraping of the leads just as you would testing any other speaker.

7. Mount them. I really want this to be attached to a glove or something neat like that, but for a proof of concept a piece of wood will do. I traced my fingers and drilled holes to fit the spare end of the tube where my fingertips would go.

## Step 2: Building the Circuit

First, a note of caution: this is pretty much the first circuit I've ever put together myself. Exactly how close to correct it is I cannot say, just that it is close enough! To my knowledge, nothing has yet been damaged--all magic smoke remains tentatively contained. Any comments regarding potential problems, improvements, etc. are absolutely encouraged and appreciated.

With such an exciting idea, it would be silly to let a little unfamiliar territory get in the way, right? That's what I thought too. Actually, with all the amazing work that has been made openly available and easily obtainable, this proved an excellent first circuit.

Here's what you will need:
Arduino (shocking, I know)
power supply (I used a bunch of 9 volt batteries)
0.47 ohm 5 watt resistor
wires
toggle switch
LED with appropriate resistor
for each tactor:
2N 2222 A transistor
1N4001 diode
#4x3/8 screws and washers
ring terminals

Here's the big picture: this handful of tactors requires a surprising amount of juice. Using Ohm's law we can figure that at 9 volts and 20 Ohms (all tactors on @ 2 Ohms each), there will be about 500 mA of current, which happens to be about what we need for a distinct vibration that won't cause numbness. These figures are well beyond what the Arduino can handle, so we need an auxiliary power supply. We also need a way for the Arduino to switch the tactors on and off, which is what a transistor does best. Electricity flowing through a coil generates a magnetic field, and a magnetic field passing through a coil generates electricity; these are intensely useful facts, but generally we only want one of these from a magnet/coil assembly, hence we use diodes to protect the electronics from electromagnetism going in the wrong direction. The last piece of the puzzle is the giant resistor, which is used to make the tactors have the same amount of current and thus force whether one is on or all of them are.

If you don't know how to build this circuit, the original size image of the breadboard can be used for reference. Note that because we are using a second power supply, the circuit will need to share a common ground with the Arduino.

Because of the amount of current each tactor needs, it is important to have excellent contact with the winding wire, which is difficult because of its tiny area and fragility. One way to go about this is to compress it between two washers; a washer usually has a rounded side and a flat side with a rough edge. Put the wire between two smooth sides to reduce the chance of damaging it. It is preferrable to use ring terminals for the wires that will join with the circuit, because they will help distribute the pressure evenly. Alternatively, you can try to use 3 washers and just pinch the wire between two that aren't holding the winding wire.

We now have a vibrotactile HCI, ready to vibrostimulate its heart out (which, with 3 9v batteries, doesn't take long). This device could also be used to convey distance and color to the blind, simulate texture, and a whole bunch of other stuff that nobody (myself included) has yet thought of, but without software it will remain a really technical doorstop. Next step!

## Step 3: Building the Software

We're on the home stretch. All we need to do is take some sound data, perform a wavelet decomposition, and use the results to tell the Arduino when to flip each switch. This sounds like a monumental task, but once again, high praise to those who have preceded us, and for the miracle of open source and object oriented programming. With a little luck we will be able to bang some bits together to get what we need. Let's start with some of the interesting theoretical stuff.

Everybody knows that a musical note is defined by a specific wavelength, but have you ever wondered why the same note sounds different on different instruments? The reason is that within that overall wave there are a bunch of other tiny waves, too small to overwhelm the big wave but enough to change the specific nature of the wave. Thus, we hear the big wave and we say "middle C," and we hear the smaller waves in that wave and say "piano." The process of sifting out the different waves is an example of a decomposition, and it is what the cochlea does, which means it's what we need to reproduce. As mentioned previously, the cochlea is performing what is called the wavelet transformation, which was actually discovered in trying to figure out what the cochlea was doing; you could call this an example of evolution beating mathematicians to a profoundly useful concept! Now, despite being essentially done with an undergraduate degree in math, I'm barely able to understand the majority of wavelet transformations on my own, so fortunately for this project there's a mysterious black box, the jwave library, which can be abused for our purpose. We will be taking some raw sound data, sending it to jwave, (some magic happens), then receiving some data which looks nothing like sound. We will then turn that data into frequencies that match the operational frequencies of the two kinds of touch receptors (50 to 250 Hz), and finally, turn those frequencies into binary signals to be sent to the Arduino, which simply switches whatever tactor it is told to. Conveniently, the substantial disparity between neural frequency and even a low USB frequency means that we will be able to signal each tactor one bit at a time with plenty of room to spare: for USB at 9600 Mhz, we would be able to signal roughly 38 million tactors.

If you don't feel up to stitching together the code yourself, the source, an executable jar, and the openSCAD code/STL files for the 3D objects can be found over on thingiverse here . If you just want a program ready to run, download the executable, put the sketch on your Arduino, and be sure to read the README. Also, be warned this is far from production quality code.

## Step 4: Completion

With all the pieces in place, there's just one last step: load up some music and let her rip!

The first question people ask is probably the one you have in mind right now, "does it work?" There are a few ways to answer this.
Answer 1: Build it and find out!
Answer 3: Yes, in a sense. First, remember that each cochlea uses about 3600 neurons to convey sound. Comparing them directly, 8/7200 is about 0.11%, which is a really low rank approximation. I made some attempts to improve on this through my particular use of the wavelet transformation, but without significantly more tactors there's no way this will come close to healthy hearing. However, early in thinking about this project I decided that a good criteria for success would be the ability to distinguish music from speech with this device; so far, all but one person has been able to do that. Another person, a musician, was able to pick out the higher structures of music. Actually listening to music in addition to feeling it is a very interesting experience, like a tactile version of a visualizer, and any doubts disappear; it rapidly becomes clear that there is a direct correspondence between the sound and what is happening at your fingertips.

Suppose you had no hearing, would .11% more ability to hear be useful? I'm going to say yes, and probably more than just .11% more useful. One of my greatest motivations for this project was the possibility of enabling somebody without hearing to experience music. To be certain, there's a lot of technology out there that holds a lot more promise to restore hearing in a conventional sense. I think this is fantastic news, and takes nothing away from me or this device.

Another significant motivator was the chance to do something that nobody has ever done before, which is to say something extraordinarily creative, something to challenge and extend my skillset. I'm still not sure if anybody else has done this, but I can say with absolute certainty that this pushed and tested my abilities; I deeply hope that this is only the beginning of the innovations that come from my life, but I must admit I'm so impressed by it that I have a hard time seeing how I could top this. Naturally, it is difficult to say what I would do with a laser cutter (besides engrave everything in sight). What I can say, is that like all tools, it would extend my ability to transform imagination into reality, even extend my ability to imagine, and I'm interested to see what could come of that... aren't you? Actually, the same is true of all the other prizes too, if to lesser degrees.

Participated in the
4th Epilog Challenge

## Recommendations

135 18K
6 341
90 7.7K
43 6.5K

• ### Home and Garden Contest

Actually this has been done before. Its called a Neurophone, invented by Patrick Flanagan when he was 13 years old in 1958. He has a patent on it, and the only reason he got it was because the patent officer was deaf and Patrick hooked up his Neurophone to him and the officer listened for the first time in his life. The officer cried and approved his patent.... But!! good job reinventing it! This took very difficult out of the box thinking that only someone of a higher level of thinking could do. Great Instructable!!!

This looks awesome. I think you mention at the beginning that you got ideas from other instructables (or at least you've looked at others) - could you provide some links? Maybe something simpler that an amateur could start with before working up to your instructable? Thanks!

Thanks! I was speaking in general, there actually wasn't anything specific I was building off of. To start somewhere simpler, I suggest looking into winding your own electromagnet and any basic Arduino projects. Since the code and circuit is provided, that pretty much covers it; despite how complicated it seems it's fairly simple to build.

even if its not original, its an impressive re-realisation of a concept, you should give yourself more credit (without going O.T.T of course, your humble personality is pleasant, thanks to the appeal to the general arrogance of myself and the majority of the human race) about the washers, a better way is to start with a piece of flat material the thickness you want, and a hole saw with the inner diameter of the washer size drill or flycutter with blades spaced for the inner and outer diameter. for a cheap flycutter you can use a bit of steel or aluminium silver-soldered or welded or turned as a single piece to a shaft that can fit in the chuck of your drill(-press preferably) and sharpen the shanks of dead drill bits into blades and bolt another piece of thinner steel with a sliver of wood in between to the first bit so that you can adjust their position and then lock them into place against the wood (or lead or copper or other semi-hard material)

Hi :) interested in exploring this myself but probably because im a total noob at Java I don't get how the code in the files was used by java to create the wavelet decomposition. Can you possibly put more detail in that section? Totally think the idea is amazing :) great work

Thanks for the compliments. I can go into more detail, but specifically what detail depends on what you know/how much work you want to do. If you've never done OOP, or if JDK, JRE, IDE, and SDK are mostly meaningless to you, then it would probably be best for me to upload a version of the program that you can just double click to run (if I can get such a thing to work). Otherwise, acknowledging that my code is a mess, the decomposition/transformation is performed by passing chunks of the appropriately structured data through iterative method calls to a Transform object from the jwave library, in essence:
Transform t = new Transform(new AncientEgyptianDecomposition(new FastWaveletTransform(new Daub02( ) ) ) );
decompositions = new double[nSamples][nDivisions];
for(int i=0; i<nSamples-1; i++){
for(int j=0; j<nDivisions; j++){ samples[j]=scaledAudio[marker++]; }
decompositions[i]= t.forward( samples ); //the transformations happen here
}

OK that does make more sense now. I should get my friend who programs Java to help :P playing around with the idea of increasing the number of actuators. Which would mean modifying the code. Way out of my league, shouldn't be a problem for my friend though. Have you thought about having more than 10 actuators?

I've definitely thought about having more tactors, the biggest hangups for me were multiplexing the signals and the increased power requirements. If I were to do more, I would definitely try to make tactors with smaller diameter magnets and seek to reduce the noise they generate, normalize magnet travel (as in being attached to the membrane), etc. I would also revisit the mounting, because having 8 mounted to a board is awkward enough; actually one of the better ideas I had in this regard was building them into a ball. Unfortunately, the code would need to be modified a little more extensively than changing the right numbers here and there, but hopefully your friend can figure it out based on some of the comments I left in the code. One trick I did in hopes of getting better results was asking for a higher dimensionality and only using parts of it--otherwise, most of the signal would be sent to one tactor, which would always be active, and all the others would just have the occasional blip. Let me know how it goes!

One idea I had for increasing the number of tactors is to use a matrix system for turning them on and off. the idea is that the board will only turn one tactor on or off at once but it cycles though them all so quickly that you cant tell the difference. Same as a computer screen or tv. Im guessing that that would take a Real lot of modification though :P

Cool project. Reminds me of some work done at MIT back in the 90's...
(see http://www.rle.mit.edu/media/pr142/23_Reed.pdf )

Oh, the irony. I've found a reference for another paper by Jacob Kirman dating back to 1973 with the same general concept and the same title as the MIT paper. Apparently I was insufficient in searching for references, and it would seem even this mistake in this specific context has a precedent. An interesting coincidence, that one.

Awesome, I can't believe that I didn't find this!! You really would think that searching for the word "vibrotactile" would significantly narrow the results to some very specifically relevant information. Thanks for sharing this, I'm far happier to have the question resolved than I am let down by not being the first. On the other hand, jeez, are there any original ideas left!? Well, at least attempting to distinguish what I think I know from what I really don't know pays off again.

I'm honored to be on the front page, it satisfies me to know that I am capable of producing something agreeable. Thank you.

If this system can be improved so the definition between sunds is better think of how it could help the deaf!

Yup, with a little bit more than common materials cobbled together by a hand tool wielding mathematician this could easily scale, and hopefully the effect would scale as well. Scaling would also enable the simulation of virtual textures, so, for instance, you could actually feel a surface in a video game. Like I said in the last part, there are more promising, conventional avenues for overcoming deafness (ie cochlear implants, which are similar in principle), but this does have the advantages of low cost and no surgery. There would be many challenges though; a major one would be moving it off the hands for maximum usability, but the distribution of touch sensing neurons is much lower around most of the rest of the body, so the resolution would be limited. Regardless, I think that this device as-is may well provide sufficient stimuli to give a deaf person a substantial portion of the experience of music. I wish I knew a deaf person so I could find out!