Haptic Feedback Device for the Visually Impaired [Project HALO]





Introduction: Haptic Feedback Device for the Visually Impaired [Project HALO]

Humana Health Challenge

Grand Prize in the
Humana Health Challenge

I recently watched an episode of Stan Lee’s Superhumans which featured a blind
man who used a series of clicks, like a bat, to echo locate his surroundings. I
got to thinking about other blind people and their ability to navigate freely –
without the use of a guide dog or cane. I came up with the idea to use a series
of rangefinders that would take input from sensors and output feedback to pulse
vibration motors placed on a person’s head. As a person gets closer to an object
the intensity and frequency of the vibration would increase – it’s directly
proportional to the distance of an object. If a region was lacking feedback,
then it would be safe to proceed in that direction.

I call my submission the H.A.L.O. - the Haptic Assisted Locating of Obstacles. I
believe this can serve very useful for the visually impaired to have the freedom
to possibily move about hands-free without the assistance of a cane or seeing
eye dog. Technology has undoubtedly made our daily lives better. By using a few
inexpensive components and sensors, I’ve made a device that will allow the blind
to navigate their surroundings and avoid collisions.

Step 1: Overview and Parts List

Major Build Portions of the Project:
- Building the Halo
- Building the Motor Modules
- Building the Haptic Headband
- Wiring the Controller
- Creating the Software

The following is the parts list that will be relevant in the subsequent steps of this Instructable.  

 - Rigid frame (I used a round embroidery frame)
 - Female headers (for the sensors)
 - Ultrasonic Rangefinders (Parallax Ping Rangefinders)
 - Wire (Wires with male and female leads are convenient)
 - Glue
 - Twist ties to tidy up wiring
 - Soldering station
 - Male headers (for creating a bridge to feed 5v and ground
 - RJ-45-Term Screw Terminal (2)
 - RJ-45 Cable
- Marker

Motor Modules:
 - Vibration Motors (5) - Motot, VIB,3V/60mA, 7500RPM
 - Grid-Style PC Board
 - Male header pins
 - Motor "shroud" (to prevent things getting sucked into the motor)

Haptic Headband:
 - Headband
 - Sewing Kit
 - 5 Motor Modules
 - Wire (Wires with male and female leads are convenient)
 - Safety Pins
- Female headers
- Soldering station
- RJ-45-Term Screw Terminal (2)
- RJ-45 Cable
- Marker

Wiring the Microcontroller:
 - Arduino Mega 2560
 - Wire (Wires with male and female leads are convenient)
 - 5 LEDs
 - Darlington IC - ULN 2803A
 - 2 port screw terminal
 - 9v battery
 - 5v regulator

Building the Software:
- USB cable
- PC (for editing code and downloading to Arduino)
- Arduino
- Arduino development environment (www.arduino.cc)
- Source Code, modified Ping.h library 

Step 2: Building the Halo

There were a couple of key considerations for the Halo (sensor) portion
of the apparatus. It needed to be rigid in order to reliably range find
the right regions of the space relative to the user facing. I determined
that 5 sensors would be a good number between being overloaded with
information, and lacking sufficient detail and there being gaps in the
field of "vision".

Full Left (-90 degrees)
Left Center (-45 degrees)
Center (0 degrees)
Right Center (45 degrees)
Full Right (90 degrees)

1- Mark your frame at appropriate locations with a marker.
2- Cut female headers to 3 pins size (these are the receptacles for the ultrasonic sensors)
3- Glue headers to the frame at appropriate positions
4- Solder all the ground wires together (these are the left-most pin looking in to the frame)
5- Solder all of the 5V wires together (these are the center pins)
6- Run individual wire to each signal pin (right most pin)
7- Twist tie loose wiring to frame
8- Terminate all wires into a RJ-45-TERM. This will be sent over an RJ-45 cable to the micro controller for processing. 

Step 3: Building the Motor Modules

The Motor Modules create the vibrations against the skin to serve as the
haptic feedback. These motors convey the distance to the objects by
vibrating more intensely and in shorter intervals.

1- Cut your PC Board into small strips (enough for the 2 male pins to be soldered on). You will need 5 of these. I did this with my bandsaw.
2- Cut the male headers into 2 pins. You will need 5 of these pairs
3- Solder the male header pairs onto the PC board
4- Solder the motor leads to the PC board (direction is not critical)
5- Cover the motor in the “shroud”
6- Glue the PC board to the shroud, with the 2 male pins facing up 

Step 4: Building the Haptic Headband

The key considerations for this were fairly simple. It had to be
flexible to fit to various user's heads. It had to bring the motor
modules to close proximity to the skin, and it had to minimize the
transferrance of vibration to avoid location confusion.

1- Put your headband on and mark the 5 key locations aligning with the sensor halo (Left, Center Left, Center, Right, Center Right, Right)
2- Sew 5 small pockets into the headband, large enough to receive the Motor Modules. These should be placed at the marked locations.
3- Cut the female headers into pairs of 2. This is to send the common 5V to the next and previous motors.
4- Affix the female headers to one of the male pins on the motor module.
5- Place the motor module into the pockets of the headband
6- Run wire to link the 5v common signal
7- Run individual wires to each motor module signal pin
8- Safety pin wires together and collect in back
9- Terminate all wires into a RJ-45 TERM. This will be send over an RJ-45 cable to the microcontroller for processing. 

Step 5: Wiring the Microcontroller Breadboard

This section covers all the connections for the microcontroller and the main breadboard. I will not describe this textually, because I am providing the files for you (REFER TO WIRING DIAGRAM IN STEP 1- created with Fritzing). However, I will hit some high points and key concepts for you.

Key Topics:

Microcontroller Selection:
The Arduino Mega 2560 was selected because of the additional PWM pins it provides.
Please refer to arduino.cc for information about the HW, or the Integrated Development Environment (IDE)

Sensor Input:
- 5 of the Pulse Width Modulation (PWM) pins will be used to read the range values form the sensors.
- The input pins are coming in from the RJ45-TERM from the Halo.
- 3-7 are used for this purpose

Motor Output:
- 5 of the PWM pins will be used to send pulses to the motor through the Darlington IC. This IC connects a load to ground when and input (these 5 pins) are asserted.
- Pin 12 – Pin 8 are used for this purpose
- The output pins (its mirror across the body of the IC) are connected to the terminations of the RJ45-TERM heading off to the headband motors
- The “first stop” for these outputs from the PWM pins are the LEDs used in our debug array. Makes pretty blinky demos too!

Power Subsystem
- A 9v battery is used to power this configuration so we don't requre a wall plug
- The 9v battery terminals are wired to the Arduino Mega Vin and GND inputs, so this provides power to the microcontroller
- A 5v regulator is connected to the 9v terminals and this is sent to drive the Darlington IC (and in turn, the motors) so we have 2 power systems and the Arduino is isolated
- A 2 port screw terminal is used to receive the battery terminals 

Step 6: Creating the Software

Being a software engineer, I spent a lot of my time on the software aspect of this project. My source code is available. I used Caleb Zulawski's Ping Library (http://www.arduino.cc/playground/Code/Ping). I did make one modification, however. This library uses a default timeout of the pulseIn() function of 1 second. This was causing large delays in the execution of the program so I reduced this timeout to 500ms. Things execute far faster now. I will not go into the details of the program, because not all user of Instructables are code-jockeys, but here are the main points:

Source Code (Arduino Sketch and modified Ping library) are at  http://polymythic.com/HALO_SourceCode_v8.zip

The flow of the main program loop() is:
- Fire sensor,
- Check to see if any of the motors are supposed to turn on or off based on previous range finding
- Fire next sensor.. Repeat

Other Things to Note As You Look At The Code:
There are 4 "intensities" of motor pulsing to give the person a better sense of the range, and these vary by the foot (up to 4 feet)
This is based on an state-machine model, but one of the Arduino threading libraries could be used to handle this independently.

Step 7: Taking It for a Test Walk

This was very gratifying.  We are all untrained users, and it does take some getting used to, but we all improved quickly.  It is hard to tell our brains to head in the direction where there is no buzzing when there is something buzzing.  That haptic buzz calls out for attention.  It was impressive to see my friends grasp what was going on, start walking with more confidence, and move around a room effectively with absolutely no vision.  As you can see, blindfolds were applied.  Note the end of the video when a chair is even detected.

Step 8: Wrapup/Conclusions

When I demoed the H.A.L.O. for a few friends and let them try it out (friends
who were seeing it for the first time and didn't really know what to expect) the
many uses for this device began to take shape - from navigating hallways to
being able to walk while carrying something to learning the lay of the land in a
new place. I was excited to see my friends walk blindfolded unencumbered and
collision free (please check out the video). It also made me think of various
modifications that I could make to the H.A.L.O.

What I’ve made is clearly a prototype and the components could be hidden in a
ballcap or visor in future models so it’s more user friendly. Additionally, the
version I made only has sensors 180 degrees around the head – you could increase
that to 360 to receive feedback in all directions or increase the sensor
sensitivity for farther distances.

I had a lot of fun working on this project and learned a ton. When we look at
everyday objects differently or truly think about ways we can use technology to
improve our lives it's amazing what we can come up with. I welcome your thoughts on this project. Good luck tinkering!



    • Pocket-Sized Contest

      Pocket-Sized Contest
    • Pro Tips Challenge

      Pro Tips Challenge
    • Science of Cooking

      Science of Cooking

    We have a be nice policy.
    Please be positive and constructive.




    Hi polymythic, I'm so happy to read this instructable.
    I worked on a very similar project last year, for the Intel STS competition. It also uses 5 sensors and vibrating motors to convey distance information to the user. I like your idea of placing the motors around the head - I put them on a belt, against the lower back. I suppose that makes sense because whereas you have the sensors spread around 180 degrees, mine all face forward at slightly different angles, so the feedback doesn't really need to wrap around the body. The sensor placement is an interesting design choice - I wonder whether it's more useful to have good resolution in front of your face or be able to sense things in all directions around you. I guess if you added enough sensors, you could have both! Too bad vibrating motors are so imprecise...
    I haven't had chance to check out your code yet, I'm looking forward to it.
    Best of luck, I'll be voting for you.

    Great to hear from you emattrose. Sounds like you've been down this road! The sensor layout I selected because I wanted to be like "natural" vision with some peripheral. Someone else had commented on making the front stronger and peripheral weaker, which is interesting. The vibration sensors are a bit imprecise, but with the right material to minimize mechanical transferrance, you should be able to get some good resolution on the haptic. Hmm.. Perhaps one of those beds that you jump on and the next person doesnt feel it. The code still has some artifacts of my free play, so feel free to do whatever. Also I have heard of a threading library for Arduino which could have made my life easier than this crazy leapfrogging state machine with timeouts and scheduling.

    Great Job poly! Breaking new ground with something we've had access to for a long time. Such a great idea. I think using this with an assist from a can for the low stuff, curbs and the like, the blind could really speed up and move more freely in unfamiliar places. Really good idea. Good luck with this. Benmansfield's advice to hit up a sight center sounds right on and could really help refine the design and bring up some 'real world' issues seeing impaired have that sighted people aren't even aware of. Again... good luck man!

    Thanks for the feedback. Agreed that the curb and low angle stuff is another part of the solution that should be integrated into the haptic picture. This project and people's commentary has fired up the mental gears thinking about assistive tech ideas.

    I think this is the coolest thing I've ever seen on Instructables.

    I think it'd be great to do a glove which you can feel distance on, too!

    Also, would be great for able-bodied people, eyes in the back of the head...

    Wow, bishopdante. That is high praise considering some of the AMAZING things that come up on this site. There are many directions to take this project for sure. Its a different experience having vision and using the device vs. being blindfolded. Your brain really does reach out for some data when you are blindfolded.

    I seriously made an account just to tell you how wonderful this is... i honestly could see this being used by visually impaired people around the world. The idea is astounding. maybe a suggestion would be to put motors elsewhere as well as the head, it seems like it could get a little crowded up there with all the buzzing and im sure you have considered this, but still its such an astounding idea.

    Thanks so much for your nice words. You are right in that if there is a lot of detection in an area, multiple adjacent sensors are firing. Teasing out the bit of quiet in the "hallway" situation (left and right firing, forward is no vibration) requires some focus and training. However, it is amazing that as a sighted person, when blindfolded, you quickly focus your brain computing on those stimulus. I would like to see how I do after being blindfolded with the rig for an hour or so.

    Easily one of the most useful ideas on this site. I could definitely see this changing hundreds of lives. Possibly extend the range, and use a variable-vibration motor(if it exists), so that you can 'see' things across a room, and the vibration gets consistently stronger as the distance narrows. Also, perhaps if you used electric pulses instead of vibrations, the hardware could be slimmed down considerably. This is definitely worth patenting. I can also think of several military applications for a cheap, easy to use, set of night vision goggles that offer 360° vision.

    Thanks for the comments, comander01. One of my design objectives was to keep the project at a resonable cost, but far more could be done with different sensors (Kinect, anyone?) The form factor can be considerably improved, I agree. I could get this into a normal hat. I do in fact use variable vibration motor. I use the PWM output for the intensity of the motor, and frequency between pulses as well.