Haptic Feedback Device for the Visually Impaired [Project HALO]

I recently watched an episode of Stan Lee’s Superhumans which featured a blind
man who used a series of clicks, like a bat, to echo locate his surroundings. I
got to thinking about other blind people and their ability to navigate freely –
without the use of a guide dog or cane. I came up with the idea to use a series
of rangefinders that would take input from sensors and output feedback to pulse
vibration motors placed on a person’s head. As a person gets closer to an object
the intensity and frequency of the vibration would increase – it’s directly
proportional to the distance of an object. If a region was lacking feedback,
then it would be safe to proceed in that direction.

I call my submission the H.A.L.O. - the Haptic Assisted Locating of Obstacles. I
believe this can serve very useful for the visually impaired to have the freedom
to possibily move about hands-free without the assistance of a cane or seeing
eye dog. Technology has undoubtedly made our daily lives better. By using a few
inexpensive components and sensors, I’ve made a device that will allow the blind
to navigate their surroundings and avoid collisions.

Step 1: Overview and Parts List

Major Build Portions of the Project:
- Building the Halo
- Building the Motor Modules
- Building the Haptic Headband
- Wiring the Controller
- Creating the Software

The following is the parts list that will be relevant in the subsequent steps of this Instructable.  

Halo:
 - Rigid frame (I used a round embroidery frame)
 - Female headers (for the sensors)
 - Ultrasonic Rangefinders (Parallax Ping Rangefinders)
 - Wire (Wires with male and female leads are convenient)
 - Glue
 - Twist ties to tidy up wiring
 - Soldering station
 - Male headers (for creating a bridge to feed 5v and ground
 - RJ-45-Term Screw Terminal (2)
 - RJ-45 Cable
- Marker

Motor Modules:
 - Vibration Motors (5) - Motot, VIB,3V/60mA, 7500RPM
 - Grid-Style PC Board
 - Male header pins
 - Motor "shroud" (to prevent things getting sucked into the motor)

Haptic Headband:
 - Headband
 - Sewing Kit
 - 5 Motor Modules
 - Wire (Wires with male and female leads are convenient)
 - Safety Pins
- Female headers
- Soldering station
- RJ-45-Term Screw Terminal (2)
- RJ-45 Cable
- Marker

Wiring the Microcontroller:
 - Arduino Mega 2560
 - Wire (Wires with male and female leads are convenient)
 - 5 LEDs
 - Darlington IC - ULN 2803A
 - 2 port screw terminal
 - 9v battery
 - 5v regulator

Building the Software:
- USB cable
- PC (for editing code and downloading to Arduino)
- Arduino
- Arduino development environment (www.arduino.cc)
- Source Code, modified Ping.h library 

Step 2: Building the Halo

There were a couple of key considerations for the Halo (sensor) portion
of the apparatus. It needed to be rigid in order to reliably range find
the right regions of the space relative to the user facing. I determined
that 5 sensors would be a good number between being overloaded with
information, and lacking sufficient detail and there being gaps in the
field of "vision".

Full Left (-90 degrees)
Left Center (-45 degrees)
Center (0 degrees)
Right Center (45 degrees)
Full Right (90 degrees)

1- Mark your frame at appropriate locations with a marker.
2- Cut female headers to 3 pins size (these are the receptacles for the ultrasonic sensors)
3- Glue headers to the frame at appropriate positions
4- Solder all the ground wires together (these are the left-most pin looking in to the frame)
5- Solder all of the 5V wires together (these are the center pins)
6- Run individual wire to each signal pin (right most pin)
7- Twist tie loose wiring to frame
8- Terminate all wires into a RJ-45-TERM. This will be sent over an RJ-45 cable to the micro controller for processing. 

Step 3: Building the Motor Modules

The Motor Modules create the vibrations against the skin to serve as the
haptic feedback. These motors convey the distance to the objects by
vibrating more intensely and in shorter intervals.

1- Cut your PC Board into small strips (enough for the 2 male pins to be soldered on). You will need 5 of these. I did this with my bandsaw.
2- Cut the male headers into 2 pins. You will need 5 of these pairs
3- Solder the male header pairs onto the PC board
4- Solder the motor leads to the PC board (direction is not critical)
5- Cover the motor in the “shroud”
6- Glue the PC board to the shroud, with the 2 male pins facing up 

Step 4: Building the Haptic Headband

The key considerations for this were fairly simple. It had to be
flexible to fit to various user's heads. It had to bring the motor
modules to close proximity to the skin, and it had to minimize the
transferrance of vibration to avoid location confusion.

1- Put your headband on and mark the 5 key locations aligning with the sensor halo (Left, Center Left, Center, Right, Center Right, Right)
2- Sew 5 small pockets into the headband, large enough to receive the Motor Modules. These should be placed at the marked locations.
3- Cut the female headers into pairs of 2. This is to send the common 5V to the next and previous motors.
4- Affix the female headers to one of the male pins on the motor module.
5- Place the motor module into the pockets of the headband
6- Run wire to link the 5v common signal
7- Run individual wires to each motor module signal pin
8- Safety pin wires together and collect in back
9- Terminate all wires into a RJ-45 TERM. This will be send over an RJ-45 cable to the microcontroller for processing. 

Step 5: Wiring the Microcontroller Breadboard

This section covers all the connections for the microcontroller and the main breadboard. I will not describe this textually, because I am providing the files for you (REFER TO WIRING DIAGRAM IN STEP 1- created with Fritzing). However, I will hit some high points and key concepts for you.

Key Topics:

Microcontroller Selection:
The Arduino Mega 2560 was selected because of the additional PWM pins it provides.
Please refer to arduino.cc for information about the HW, or the Integrated Development Environment (IDE)

Sensor Input:
- 5 of the Pulse Width Modulation (PWM) pins will be used to read the range values form the sensors.
- The input pins are coming in from the RJ45-TERM from the Halo.
- 3-7 are used for this purpose

Motor Output:
- 5 of the PWM pins will be used to send pulses to the motor through the Darlington IC. This IC connects a load to ground when and input (these 5 pins) are asserted.
- Pin 12 – Pin 8 are used for this purpose
- The output pins (its mirror across the body of the IC) are connected to the terminations of the RJ45-TERM heading off to the headband motors
- The “first stop” for these outputs from the PWM pins are the LEDs used in our debug array. Makes pretty blinky demos too!

Power Subsystem
- A 9v battery is used to power this configuration so we don't requre a wall plug
- The 9v battery terminals are wired to the Arduino Mega Vin and GND inputs, so this provides power to the microcontroller
- A 5v regulator is connected to the 9v terminals and this is sent to drive the Darlington IC (and in turn, the motors) so we have 2 power systems and the Arduino is isolated
- A 2 port screw terminal is used to receive the battery terminals 

Step 6: Creating the Software

Being a software engineer, I spent a lot of my time on the software aspect of this project. My source code is available. I used Caleb Zulawski's Ping Library (http://www.arduino.cc/playground/Code/Ping). I did make one modification, however. This library uses a default timeout of the pulseIn() function of 1 second. This was causing large delays in the execution of the program so I reduced this timeout to 500ms. Things execute far faster now. I will not go into the details of the program, because not all user of Instructables are code-jockeys, but here are the main points:

Source Code (Arduino Sketch and modified Ping library) are at  http://polymythic.com/HALO_SourceCode_v8.zip

The flow of the main program loop() is:
- Fire sensor,
- Check to see if any of the motors are supposed to turn on or off based on previous range finding
- Fire next sensor.. Repeat

Other Things to Note As You Look At The Code:
There are 4 "intensities" of motor pulsing to give the person a better sense of the range, and these vary by the foot (up to 4 feet)
This is based on an state-machine model, but one of the Arduino threading libraries could be used to handle this independently.

Step 7: Taking It for a Test Walk

This was very gratifying.  We are all untrained users, and it does take some getting used to, but we all improved quickly.  It is hard to tell our brains to head in the direction where there is no buzzing when there is something buzzing.  That haptic buzz calls out for attention.  It was impressive to see my friends grasp what was going on, start walking with more confidence, and move around a room effectively with absolutely no vision.  As you can see, blindfolds were applied.  Note the end of the video when a chair is even detected.

Step 8: Wrapup/Conclusions

When I demoed the H.A.L.O. for a few friends and let them try it out (friends
who were seeing it for the first time and didn't really know what to expect) the
many uses for this device began to take shape - from navigating hallways to
being able to walk while carrying something to learning the lay of the land in a
new place. I was excited to see my friends walk blindfolded unencumbered and
collision free (please check out the video). It also made me think of various
modifications that I could make to the H.A.L.O.

What I’ve made is clearly a prototype and the components could be hidden in a
ballcap or visor in future models so it’s more user friendly. Additionally, the
version I made only has sensors 180 degrees around the head – you could increase
that to 360 to receive feedback in all directions or increase the sensor
sensitivity for farther distances.


I had a lot of fun working on this project and learned a ton. When we look at
everyday objects differently or truly think about ways we can use technology to
improve our lives it's amazing what we can come up with. I welcome your thoughts on this project. Good luck tinkering!