Talking Joystick Mouse

I work in Assistive Technology, which is technology designed to help people with disabilities stay independent.

We encountered a man who had quadriplegia due to a motor vehicle accident.  He had a little motion in one hand and wanted to use a computer, but he could not use a regular mouse.  If he could use the computer, he could go online, surf the internet, research whatever struck his fancy, chat with other people, email, etc. - it would open a lot of doors for him and greatly increase his standard of living.  We tried a commercially available joystick mouse, and he could use it successfully.  The problem was cost: the commercial model was $550 (since then, it has dropped to $400 ) which neither he nor his family could afford.

I built a joystick mouse for him out of a USB gamepad and arcade machine components.  This mouse had two features lacking in the commercial model: 1) It could launch programs or commands, greatly increasing the efficiency of computer use  2) it could talk, giving auditory feedback regarding which button was pressed, as he could not move his head to look at his hands. 

We have since used descendants of the first joystick mouse with several people who had cerebral palsy or other conditions that made traditional computer use difficult.  The cost to build one of these is about $45.

Step 1: Materials and Tools

Materials:
Wood, about 7" wide and 1/4" thick
Paint, Wood glue

(1) USB gamepad (I have had good success buying cheapies from eBay or other places - we will be destroying it in the process so don't buy the most expensive).

(1) competition joystick (happ.com is a good supplier too, but more expensive)
(6) competition pushbuttons (happ.com is more expensive here too)
(4) #10 1/2" wood screws (or metal screws)
(7) #8 or #6 1.5" Wood screws
(20) 0.187" Female Quick Disconnect Crimp Connectors
300" (25 feet) 24-28-gauge hookup wire (old CAT-5 cable works beautifully with the jacket removed)

12x12" square solid foam shelf liner

Tools:
Drill
Dremel with micro drill bit (e.g. 0.0260")
1.125" hole saw (for drill)
Table Saw / regular saw for woodworking

Needlenose Pliers
diagonal-cutting pliers (dykes)

Sandpaper

Soldering Iron and solder
Wire strippers
Helping hands
Wire crimp tool

Hot air gun
Hot glue gun

Software:
AutoHotkey (Windows only, similar automation software exists for Mac / Linux)

Step 2: Make the Box

The housing for the joystick mouse can be as simple or as complex as you want.  For mine, I am going to be using a wooden box that I had a friend of mine make (who is more gifted in carpentry).  Hence, I don't have pictures for that part.  My first model was built using particleboard shelving, dowels, and cardboard.  I have experimented with stiff Tupperware containers.   The templates showing the original (side view) and top-piece template are attached.

Saw two rectangles of wood 7"x8" (1/4" to 1/2" thick).  These will be the top and bottom pieces.  Determine the angle of the finished joystick mouse, then cut identical trapezoids for the sides.  The height of the joystick mouse should be as short as possible.  If you use thinner would (1/4" over 1/2" and measure to make sure you don't have a lot of wasted space inside vertically it increases the ergonomics of the device.

The front and back will be rectangles the same width but of different heights (because the back of the mouse is raised up higher).  If it fits, round the corners and use wood glue or screws to hold the sides of the box together.  If it doesn't, sand/cut/file/hammer/swear until it does.

The holes for the pushbuttons need to be 1 1/8"  (1.125").  We are going to drill seven holes in the top, one for each button and one for the shaft of the joystick. 

The top or the bottom needs to be left off until the end.  Make sure it fits now, and predrill your pilot holes for the screws.  You may even want to screw it together lightly to make sure everything works, then unscrew top and bottom until the end of the project, leaving the "box" alone.

For the rear piece you will need a hole for the USB cable to pass through the housing.  Originally, I had this hole beautifully centered on the rear panel.  The problem is that this is too small for the USB header to fit through.  This means I have to either unsolder the USB cable from the gamepad, pass the cable through, then resolder it - or clip the USB cable in half and then splice/solder all the wires inside back together.  Ugh.  Not fun. 

In the attached pictures, I ignore the nice hole drilled in the middle and make much better use of a small mouse-hole drilled in the bottom edge of the piece.  This simplifies the build process considerably.

Once things are sanded/painted and ready to go, cut a piece of solid shelf liner a little smaller than the bottom.  Hot-glue this into place.  This provides an anti-skid surface that helps keep the joystick in place during use.  You could also use the rubber furniture bumpers, but they come off easier.

Step 3: Insert the Arcade Controls

In this step, we put the arcade pieces into the top panel.  The joystick and the pushbuttons all come with good exploded diagrams and installation instructions.

For the pushbuttons:
Insert the buttons through the holes in the panel from the top.  From the back, put the nut on and hand-tighten only most of the way (the nuts are plastic, metal tools will tear them up).  Insert the cherry microswitch.  The microswitch will only snap into the pushbutton one way.  There are tabs on the pushbutton that mate with holes in the microswitch.  Turn the button+microswitch assembly so that the contacts point towards the center of the unit where the circuit board will go.  Hand-tighten the nut the rest of the way until it is snug.

If you are paranoid, a large drop of hot-glue will keep the nut from loosening.

For the joystick:
From the back, screw the joystick base plate into the wood.  Be careful to center where the shaft for joystick will be over the hole in the wood.  I like to cover the screws and edges with hot glue just to make sure it doesn't get any ideas about leaving.

Assemble the joystick.  Put the washer on the shaft, then put the collar.  Insert this assembly from the top.  From the bottom, put the button-masher part on the shaft (there is a proper direction, the smaller end should be towards the bottom of the shaft).  Push the button masher down the shaft until you can move the joystick and have it make good contact with the switches on the bottom.  Push the lock-ring into place to keep the joystick assembly together.

Finally:
Label all the switches with a Sharpie (see picture).  This will be very important when it is time to hook it all up.

Step 4: Wiring Part #1

In this step, we will make the wiring leads that will run from the switches to the gamepad.

We will make ten of these.  I normally make them all the same length, which makes them interchangeable.  This also means that the wiring will be a lot more of a rat nest inside the device than it optimally would be.  Ideally, the length would be carefully measured and you would have one length for the lower two buttons, one length for the next higher buttons, one length for the top buttons, a length for the left and right switches, a length for the up switch and a length for the down switch.  This is a lot of extra work for something nobody is going to see.

I am using old CAT5 cable, sans jacket, for my wiring.  Any thin solid core wire should work.  I take the CAT5 and separate it into color pairs or "twists" for cheap hook-up wire.  Cut a 14" length of a twist.  You should have two wires twisted together.  Untwist the ends about 1.5 inches.  Strip 1/4" of insulation from all four ends.

Put two the female crimp connectors on one end of the twist (one for each wire).  Using the crimp tool, crimp the connectors tightly onto the wire.  Test to make sure the ends are tight are will not work loose.

CRITICAL STEP: Test each finished set for continuity with a multimeter.  It should be continuous from the bare end of the wire to the metal on the crimp connector.  If one is not made correctly you will spend HOURS debugging why things are not working.  Word to the wise.

Make 10 of these wiring harnesses.

Step 5: Gamepad Preparation

In this step, we start preparing the gamepad for its new life as a joystick mouse.

First, plug the gamepad into the computer and test it out (see the later section on testing).  Go to the Control Panel (on Windows), Game Controllers.  You should see the gamepad listed.  Go to Properties.  Press all the buttons and move the direction pad and make sure everything is working.  The buttons will be numbered.  Test the buttons so that you know which button corresponds to which number.  Write this down, they may not have it silkscreened on the circuit board.  It always good to make sure it works before you tear it apart.

Unscrew the gamepad housing.  Gently get all the pieces out of the housing.  Throw away the housing.  Throw away the little buttons.  Throw away the little silicone plastic springs.  If it has pager motors (for rumble/feedback, cut those off and save them for some other project.  If you do cut the motors off, be sure to cut cleanly so the wires are not exposed.

Most gamepads have triggers (usually buttons 5,6,7,8).  5&7 tend to be on one trigger, 6&8 on the other.  Normally, you have three wires running to each pair of buttons.  One is common, and the other wire indicates which button (e.g. 5 or 7) is pressed.  For space reasons, we are only making the joystick with 6 buttons, but that is a tradeoff of space vs. functionality.  If your gamepad has 10 buttons, there is no reason you couldn't use all ten buttons.

With the multimeter, test the trigger circuits for continuity until you know which two of the three are used by #5 and which two of the other three are used by #6.  Clip  the trigger button circuit boards off and throw away, leaving the three wires connected to the main circuit board on each side.  Strip 1/4" off the ends used by #5 and #6.  Clip and throw away the wire that would have run to #7 and #8.

Some of the pictures for this step are from a Logitech gamepad and others are from one that looks very similar to the one I linked to in the first step.  The Logitech gamepads are a PAIN to adapt because they have very thin traces that are difficult to patch into.  The ones I link in the first step to are relatively easy to adapt.  The principle is the same for both.

To form a switch, electricity needs to make a complete path.  We are going to run wires from the pushbuttons to the circuit traces on the gamepad.  When someone presses the pushbutton, the gamepad controller get the signal that particular wires have touched and tells the computer that someone just pressed a button.  We will run the two wires from each pushbutton to each side of the switch currently on the circuit.

Look for test points. The gamepad I linked to at the start has bare copper points scattered throughout, one corresponding to most of the buttons.  This is a huge benefit and makes assembly much easier.  It also has areas where multiple buttons share a common wire.  This allows us to drill fewer holes (and thus fewer things to go wrong).

The basic steps are:
1) Identify the traces or test points corresponding to a particular button

2) Drill a very small hole in the board with a micro drill bit and a dremel so that it is either a) Through a test point   or  b) next to a trace (without touching it)

3) With a little sandpaper, sand through the lacquer until you see bare copper.  Solder will not adhere to the lacquer, so to join the wires we need bare metal

4) Repeat for all buttons to be adapted (in this case, the 4 directions and buttons 1-4)

Be very careful when drilling next to the traces.  It is very easy to slip with the dremel and destroy the traces.  If I am drilling next to a trace, I go for the widest point available.  This will give more surface area for soldering later and helps support the wires. Thinner traces can be torn off the board easier than thicker ones.

Step 6: Wiring Part #2

In this step, we actually attach the wires to the gamepad.

For starters, by looking at the circuit traces, I notice that the left and down share trace, and up and right share a trace.  It makes sense - one can't go up and down (or left and right) at the same time.  To simplify the number of holes to drill and solder, combine these  two wiring harnesses into one that has a combined point.  To do this, wrap one wire around the base of the other and then solder them together.  For the buttons 1-4,  notice that they have a shared point so do the same thing - four wiring harnesses hooked up with 1 point in common.  This will vary depending on the model of gamepad.

Insert a wire from one of the wiring harnesses from the back so it sticks up through the board on the circuit board side.  Slightly bend the wire so that it does not pull out immediately.  I sometimes put a drop of hotglue on the back to provide mechanical support and to keep the wires from backing out immediately.

On the top side (for through-testing pad wires): - apply the soldering iron to the area where the wire comes through the board.  After a couple seconds, apply the solder.  It should form a bead that covers the testing pad and holds on to the wire.  Remove soldering iron.  Once solder is cool, snip off the extra wire with the diagonal cutters.

For trace-connection wires: Fold the wire down with pliers.  It needs to make as much contact with the copper trace as possible.  It needs to be as horizontal as possible (parallel to the trace).   Apply the heat to the trace on the board and press the wire and the trace together with needlenose pliers or a small screwdriver.  Apply the solder while applying pressure with the screwdriver.  Once the solder is applied, remove the iron and then remove the screwdriver if it looks like it is holding together.  These are very fragile, so be careful.

Test everything.  See note below.

Repeat these steps for all wires until all the wires are successfully attached to the gamepad.

In some cases it is good to pre-tin the wire or the traces.  This can make adhesion easier.  To pre-tin, apply solder to the wire and the traces individually so that they are silver-colored.  Then press them together (with the pliers) and heat them up with the iron.  The solder should melt, remove the iron.  The pieces should hold together.

For the trigger buttons (#5 & #6),  fold the wiring harness wire ends into hooks with needlenose pliers, then bend the stubby wires for the trigger buttons into hooks.  Put a 1/2" piece of heat-shrink tubing over the wiring harness wires.  Link the hooks, then crimp into place with the pliers.  Solder the wires.  Move the heatshrink tubing over the solder joint.  Heat quickly with the hot-air gun to make a good seal.  This prevents the trigger wires from touching accidentally.

A common error that occurs on this step is when the rosin-core of the solder connects to one surface, thus providing an insulator.  For instance, the test pad will be coated in a tiny layer of plastic, and the solder will adhere to the wire, but the wire and the circuit board on the gamepad are electrically separate.  It is good to test with the multimeter before you test on the computer.  To fix this, heat up the bad solder joint with the iron for about ten seconds.  On the traces, sometimes sanding the rosin flux off with sandpaper helps also.

See the next step about testing. Going through the entire testing process after every two or three connections to the board is HIGHLY recommend. It helps in finding connections that are bad early-on.

Step 7: TEST ON THE COMPUTER

The importance of this step can not be minimalized.  Test often.  It is much easier if you can figure out that something is not soldered correctly BEFORE there are zillion wires everywhere.  Nip it in the bud.  You do not save time by doing it wrong.  This step makes sure that things are soldered correctly and you haven't accidentally knocked something loose.

If you have to unsolder the USB connectors on the gamepad, be VERY CAREFUL you put them back in the right order.  Otherwise the gamepad controller explodes and you could damage your computer (voice of experience, people).

Go to Control Panel.  Go to Game Controllers.  You should see the controller listed in the box.  Click "Properties" to test the controller.

Touch the pairs of wires together (or press the buttons) one at a time.  If the indicator light on the screen turns on (for a button) the connection is good.  If the + character moves in the box, it is a good connection for a direction.  If you have not done so, mark one of the crimped ends with a permanent marker so that you know what it does later.

When done, Click OK.  Remove the USB gamepad from the computer.

Step 8: Wiring Part #3

In this step, we connect the quick-connect ends of the wires to the proper pushbuttons.

Position the circuit board between the lower 4 buttons (that is the only blank spot inside the case) with the wires streaming towards the top.

One pair of wires at a time, connect the wiring harnesses to the respective buttons.  Remember to connect to the contacts marked "Common" and "NO".

Route the wires as best you can.  The only thing to really avoid is the joystick.  If wires run directly under the joystick they can jam the joystick or cause it to move erratically.  Anywhere else in the case is fair game.

Testing is a MUST for this stage also.  The odds are decent that something will get jarred loose.  Every few connections, hook the USB cable back up to the PC and test everything.

Step 9: Finishing Touches

Once everything has been tested again and all the wires are in place - it is time for a couple finishing touches. 

Put  a large dollop of hot glue on the backside of the top panel.  Press the circuit board into the glue to hold it in place.  This allows the device to be handled without the circuit board banging into things.

Put a small overhand knot into the USB cable right on the inside of the box.  If someone yanks on the cord, they yank against the wall of the box and not against the solder joints.

Put the top and bottom back on the box and put the screws in place.  The unit should now be fully assembled.

Step 10: Hardware Complete!

YAY!  All done.  Sorta.

At this point, the hardware is complete.  However, when plugged into a computer, it is still just a USB gamepad.  Wiggling the stick and pressing the buttons does nothing, because Windows/Mac/Linux don't natively use a joystick as a means of input (they all want a mouse).

To really make it useful, we need some software to translate joystick motion to mouse motion.  Here is where the program launching and talking parts come into play.

On to the software ...

Step 11: Software

To actually translate joystick movement to mouse movement, we use the program AutoHotKey.

There is a variation of AutoHotKey called IronAHK that is supposed to work under Linux or Mac using the Mono framework.  I haven't gotten it to work, but your mileage may vary.  There are other programs available for Mac and Linux that do system automation that should work with the Joystick mouse.

There are other programs that can do joystick to mouse conversions:
-----------
Innovation / RJ Cooper Joystick to Mouse
Can toggle buttons (drag), can set buttons for mouse, keyboard
Has acceleration (adjustable)   Has Sound effects
Suggested Retail: $19.95
Website: http://www.imgpresents.com/

Joystick 2 Mouse
Can set joystick buttons to mouse events (left click, etc.), windows keys, functions keys, alphanumeric keys, web addresses, etc, double-click, etc.
Has acceleration built in ( goes faster if hold it longer), - this can be adjusted or disabled
Price: Free, $20 donations accepted.
Website: http://atzitznet.no-ip.org/Joy2Mouse3/


Stickmouse
Acceleration turned on, not configurable,  Only Left and Right button
Price - $5
Website: Trajectorylabs.com

JoyMse
No Keyboard keys, no configuration, no drag or double-click
Exits if tray left-clicked, options if right-clicked
Price: Free
Website: http://www.deinmeister.de/jmt_e.htm

--------------------
Most of the programs listed above lack the ability to drag except to hold the button down while moving.  None of the programs listed above can talk, although some can make sounds.  AutoHotKey is far and away more configurable.  It is also free software.

1) Download AutoHotKey to the computer that will run the joystick mouse
2) Download the "Joystick Mouse.zip" file attached to this step
3) Unzip the Joystick Mouse.zip file to its own directory

4) Inside the directory should be a bunch of .wav sound files and the file "Joystick Mouse.ahk".  This is the file that runs the script and makes all the magic happen.

The file is set up for basic mouse operation. If a Joystick Mouse is built the way demonstrated, upper left is left-click, upper-right is right-click, middle left is double-click, middle right starts/stops dragging, lower left is speed up, and lower right is slow down.

The basic logic that the program follows is this:
Wait for joystick button press

      When a joystick button is pressed:
            Play the corresponding sound (e.g. say "Left click")
            Do the corresponding action (e.g.  click the left mouse at the current location)


     When the joystick is moved:
             Move the mouse cursor correspondingly




If you need to edit it, I would first get familiar with AutoHotKey.  It is a very powerful scripting language and I am continually amazed at how useful it can be.  There are instructions throughout the file detailing how to change and customize it.  With it, you can launch program, play sound files (which is how we get it to talk) and do anything you could do with a mouse or keyboard. 

For example, the original user didn't need the double-click or the dragging feature.  Instead, we set up the software to launch Microsoft Word when the right middle button was pressed and a talking on-screen keyboard when the left middle button was pressed.  This allowed him to quickly launch a word processor and an onscreen keyboard so that he could communicate with others and spell what he wanted or needed.  The beauty of this approach is that it is all done in software.  Using AutoHotKey, someone can tie any combination of keyboard or mouse actions to a joystick button.  This system is infinitely configurable for individual needs and preferences.

5) To record new sounds:
a) Plug in a microphone, use the Sound Recorder accessory (or Audacity)
-or-
b) Go to AT&T's Text-To-Speech Site.  Type what you want said in the box, then click "Download" to get a wav file from their site.  Store this file in the same folder as the "Joystick Mouse.AHK" file. 

6) If you need the joystick mouse to start when the computer turns on, put a shortcut to the file in the Startup folder under the system tray.  I often put a shortcut link to the "Joystick Mouse.AHK" file in situations where it is a shared computer - that way it can be turned on as needed.


If the software is running, you should see the green AutoHotKey icon in the system tray.  At this point, you should be able to plug the joystick in and steer the mouse cursor around by pressing the stick and press the buttons to left,right,double click.

Step 12: Other Applications: Button Launcher

Here is another application of the Joystick mouse.  It is a spinoff project, called "The Button Launcher."  Under the hood, it is the same as the joystick mouse, but without the joystick.

We worked with a woman who was in her 80's and had never needed to work a computer before.  Her hands were a bit shaky due to age.  Her grandchildren would only talk to her through email (rather than phone or letters or visits), so she found herself suddenly needing to use a computer.  

Unfortunately, many day-to-day operations on the computer were difficult for her. 
To check email:
1) Turn computer on.  Pretty easy
2) Once it has loaded desktop, drag the touchpad until it is over the icon that says "internet".
3) Double-click the icon.  AAARRGGH!!!!  If she moves while double-clicking, it relocates the icon somewhere on her desktop.  If she double-clicks too slow it renames it.
4) Ten minutes later, the icon has been successfully double-clicked.  Go to address bar and type "gmail.com" (she wanted google search as home page).  Hope that she types it right and doesn't misspell anything.
5) Type in username and password.  Again, since hands are shaky, odds are good that she will mistype name and/or password.  The password box is blanked out and she can't even tell if she is typing it right.
6) Click "Log in"

On average, it was taking her about 20 minutes of struggling to get her email open.  This was highly frustrating to her.  Hence the button launcher was born.

It has six buttons, each of which is scripted through AutoHotKey to perform different tasks.  Since AutoHotKey can automate everything on a computer, she buttons to do the things she wanted to do on a computer. 

Internet?  Press one button and the internet loads. 
Email?  Press one button and it goes to gmail and logs in for her. 
Word Processor? Press one button.
Solitaire?  Press one button.
Close current program?  Press one button.
Shut down PC? Press one button.

This gave her the ability to use her laptop easily, to do the things she wanted to do, and cut out all the in-between steps.  Simple.  Easy.  Efficient.

The attached file has the AutoHotKey file and all the sounds we used for her.  Instructions for altering the file for different sounds or actions are in the comments of the AHK file.

Hook up the button launcher to the PC.  Run the Button Launcher.ahk file.  All the buttons should be live and do things when pressed.