Introduction: HandSight: a Glove for the Blind to Feel Shapes and Navigate Obstacles

Picture of HandSight: a Glove for the Blind to Feel Shapes and Navigate Obstacles


HandSight is a prototype glove to aid the blind. It can sense the lightness or darkness of a surface with tactile feedback from a vibration motor for each individual finger. It can also sense distance from physical objects or obstructions and indicate direction and distance with the same vibration feedback. It supports additional modes, and the possibilities are nearly endless. The glove can connect over Bluetooth to switch modes and visualize the sensor readings.

This instructable was made as part of the final project requirement in the CS graduate course "Tangible Interactive Computing" at the University of Maryland, College Park taught by Professor Jon Froehlich. The course focused on exploring the materiality of interactive computing and, in the words of Hiroshii Ishii, sought to "seamlessly couple the dual worlds of bits and atoms." Please see http://cmsc838f-f12.wikispaces.com/ for more details.

See our Wiki class page for the project here (where we talk about some of our challenges and limitations):
http://cmsc838f-f12.wikispaces.com/HandSight

Step 1: List of Materials

- Arduino Pro Mini-328 (5v, 3.3v may work but not tested) (x1)
- IR Reflectance Sensor - QRE1113 (x4)
- Vibration Motor ROB-08449 (x4)
- Ultrasonic Range Finder (x2) http://www.amazon.com/Ultrasonic-Module-HC-SR04-Distance-Arduino/dp/B004U8TOE6/ref=sr_1_1?ie=UTF8&qid=1355861471&sr=8-1&keywords=Ultrasonic+Range+Finder
- Bluetooth Mate Silver WRL-10393 (x1)
- 68 ohm Resistor (x1)
- 9v Battery and Battery Case with Switch (x1)
- FTDI cable (for programming) (x1)
- Double Sided Sticky Tape, or other adhesive to hold the ultrasonic sensors in place
- Wire, electrical tape, and soldering materials. Rainbow ribbon cable is recommended for easy wiring of the fingertip sensors.
- Glove that you don’t mind repurposing permanently
- Extra cloth that matches the glove
- Velcro, Sewing Materials
- Windows Phone to run our visualization and control app, or any device that supports the Bluetooth Serial Port Protocol (SPP) if you want to develop your own app

All electronic components available on sparkfun unless otherwise noted

Step 2: 3D Printing

Picture of 3D Printing

Holders for the sensors can be found here:

http://www.thingiverse.com/thing:38133
http://www.thingiverse.com/thing:38134

We printed with MakerBot Replicator 2. Download STL files, open in a program called ReplicatorG. Put on platform/center. Generate Gcode.

Material PLA

Settings used:
infill% 10
layer height 0.27 mm
number of shells 1
feedrate 41 m/s
travel feedrate 56
print temp 226

plastic diameter 1.75 mm
extruder nozzle diameter 0.4mm

(in the actual gcode we changed platform temp to 060, which worked well for PLA)

Print four sets of the fingertip case, and two of the ultrasonic slot.

Step 3: Assembling the IR Sensor Package for Each Finger

Picture of Assembling the IR Sensor Package for Each Finger

Thread wire through the holes, place sensor next to them so that pin 1 is in bottom left. Carefully solder the wires to the pins, and check to make sure there aren’t any short circuits. Add the vibration motor into its hole, and solder wires to the leads so that they are the same length as the IR sensor wires. Repeat this process for the other three sensor assemblies. We used a rainbow ribbon cable for this process so that the wires stayed compact and so that it was easy to identify which wire was which.

Step 4: Attaching Fingertip Assemblies to the Glove

Picture of Attaching Fingertip Assemblies to the Glove

Cut a hole the size of the face of the 3D printed assembly into each of the four fingertips (not the thumb. Also cut a small slit on top of each finger of the glove large enough to pass through all six wires. You should reinforce the cut areas by either sewing tight stitches around them or by melting them depending on the glove’s material. Now thread the wires through the hole in the fingertip and out the hole on top of the knuckles. Repeat for the other three sensor packages.

Step 5: Attaching Ultrasonic Sensors to the Glove

Picture of Attaching Ultrasonic Sensors to the Glove

Sew the two ultrasonic sensor supports to the top of the glove, far enough back so that they won’t bend with the fingers. They should be angled slightly away from each other so that their field of view is widened (5-10 degrees). Place double sided sticky tape or some other adhesive on the plastic between the slots, and attach the two ultrasonic sensors with their power and communication pins facing up and their audio sensors facing forward. Either solder wires to the pins, or use a less permanent solution as we did (see image).

Step 6: Adding a Battery Pack

Picture of Adding a Battery Pack

Sew a pouch for the battery pack onto the back of the glove. This pouch should be wide enough to hold the battery and allow access to it at one end so that the battery can be replaced and the power can be switched on and off. Use velcro to hold the battery in place within its pouch.

Step 7: Soldering Connections

Picture of Soldering Connections

In our images, the wires are as follows:

- Blue: Power for the IR led (sensor pin 1)
- Green: Ground for the IR led (sensor pin 2)
- Yellow: Collector for the IR detector (sensor pin 3)
- Orange: Emitter for the IR detector (sensor pin 4)
- Red: Power for the vibration motor
- Brown: Ground for the vibration motor

Replacing these colors with the corresponding wires in your setup, separate the ends of the ribbon cable and solder together the grounds (green, orange, and brown). Solder each of these wires from each of the four packages together into a single connection, with an additional wire running out for connecting to the board. Also solder the four blue wires together in the same manner. Wrap all of the connections with shrink tubing or electrical tape to protect the connections and prevent short circuits.

Step 8: Preparing the Arduino Pro Mini

Picture of Preparing the Arduino Pro Mini

Solder angled headers to the six programming pins on the Arduino Pro Mini (see picture). You will use these headers to program the board over an FTDI USB cable, and they will also be used to communicate with the bluetooth module.

Step 9: Soldering Pins on the Arduino Pro Mini

Picture of Soldering Pins on the Arduino Pro Mini

Use the circuit diagram pictured to solder connections between the sensors/motors and the Arduino. The connections we used in our code are as follows:

- Blue wires (from IR LEDs) to VCC through 68 ohm resistor
- Green/Orange/Brown wires (grounds) to GND
- Yellow wires to analog pins A0, A1, A2, A3 (in order from index finger to pinky)
- Red wires digital (PWM) pins 3, 5, 6, 9 (in same order)
- VCC from two ultrasonic sensors to VCC
- Ground from two ultrasonic sensors to GND
- Trigger and Echo from left ultrasonic sensor to pins 10 and 11, and from right sensor to pins 2 and 4 (respectively).
- Red wire from battery to RAW
- Black wire from battery to GND

Step 10: Adding Bluetooth Module

Picture of Adding Bluetooth Module

Attach the bluetooth module to the programming headers on the Arduino Pro Mini. With both boards facing up, the pins line up exactly. You can use angled headers on the bluetooth module. We didn’t have female headers available, but adding them to the bluetooth module will make connecting the two sets of headers much easier. Don’t make any permanent connections, because you will still need to use the programming headers to actually program the Arduino. Note that it has been shown possible to program the board over bluetooth, but we didn’t pursue this possibility.

Step 11: Programming the Board

Picture of Programming the Board

Connect the Arduino Pro Mini to the FTDI cable, being sure to line up the green and black wires to the Grn and Blk pins correctly. Then attach the USB end to your computer, and upload the code to the board using the Arduino software.

Our code is available at:
https://github.com/lstearns86/handsight

Download the git repository, and upload the file handsight_arduino/handsight_arduino.ino.

Step 12: Connecting Via Bluetooth

We have developed a Windows Phone 8 app that can be used to control the device and visualize the sensor readings. If you don't have access to a Windows Phone 8 device, or if you would like to develop your own control/visualization program, you can do so over the Bluetooth Serial Port Profile (SPP). Simply pair the bluetooth module with your phone or computer, then open a serial port connection on the appropriate COM port. The data from the device is in the following format:

[mode]:[+|-|=][reading]|...[+|-|=][reading]|[text]

[mode] is a number from 0-5 indicating the current mode
[+|-|=] indicates whether the reading is above the detection threshold (+), between the detection and white thresholds so black is detected (=) or below the white threshold so white is detected (-).
[reading] is a number from 0-1023 for the first four readings, indicating the voltage from the IR sensor, while it is from 0-200 for the fifth and sixth readings indicating the distance measured by the ultrasonic sensors (in cm).
[text] is only present in typing mode, and gives the characters detected since the last serial update

The following commands can be sent to the device:

0: edge detection mode
1: shape detection mode (black detection)
2: grayscale mode
3: navigation mode
4: typing mode
5: massage mode
10: calibrate the detection thresholds using the current readings
11: calibrate the black thresholds using the current readings
12: calibrate the white thresholds using the current readings

Note that the numbers are the byte values, not the ascii character codes for those numbers

The Windows Phone 8 was developed in Microsoft Visual Studio 2012 on Windows 8, and is available in our GitHub repository linked in the previous step if you would like to use it.

Note that you will need to change the name of your bluetooth module to match ours, or change the C# code that connects to it.

Comments

OrpheusP (author)2016-03-10

Sir. May I know at what line should I edit the code for the bluetooth module? I have the HC-05. I can't seem to find it in the C# code.

bindassbasanta (author)OrpheusP2016-04-24

search for "handsight" and convert it for what u desire :) simple :D :D

lstearns86 (author)OrpheusP2016-03-10

The bluetooth module we used allowed us to replace the built-in serial port functionality that is usually used for USB. If you're using a different module, you'll need to find a tutorial for how to set it up and then replace our Serial.write lines with the equivalent for your module (usually there's a library you can pull in that has a nearly identical function, so a simple find and replace should suffice).

jlord11 (author)2016-03-07

Should I download the libraries of the sensor? I can't upload the sketch on my pro mini.

lstearns86 (author)jlord112016-03-10

I'm not sure exactly what problem you're encountering, but it's been awhile since we wrote this code and it's likely that some of the libraries have changed in the meantime. Please make sure that you're set up to install some other example on your board (e.g., blink), and that our code compiles. Please feel free to post any compilation or uploading errors and I'll see if I can help you resolve them.

jlord11 (author)2016-02-28

Guys I don't know how to make an app, is there any remedy to select the modes w/o using an app?

lstearns86 (author)jlord112016-02-28

We mostly used the app to help us visualize the sensor readings, it's not really necessary for changing the mode. The simplest solution would be to hard-code the mode that you want to use, or else change it via the Arduino Serial Port window. You'd need to connect it to a computer whenever you wanted to change the mode, though.

Another possibility is to add a push button, and then replace the Serial port reading code with a check for whether or not the button is down. If it is, then switch to the next mode. You'll probably also want to add some logic to check for the button release, or a limit on the amount of time that must elapse before it increments the mode again (otherwise it may cycle through the events too quickly).

jlord11 (author)lstearns862016-02-28

Thank you sir

sltower (author)2016-02-02

We have the fingers of the glove working, the adruino programmed and the bluetooth connected to a windows phone with the app installed. We can't seem to get the app to read anything. The bluethooth keeps disconnecting once it returns to the app section. The red light hasn't stopped flashing on the bluetooth. Does this mean that it is not connected to the adruino or the phone?

lstearns86 (author)sltower2016-02-02

Sound like there's a problem with the Bluetooth connection. It's been a few years since we implemented the app's Bluetooth functionality, and it's likely that something has changed in the libraries since then. Also, if you're not using exactly the same Bluetooth module as us you may need some slightly different parameters to connect to it.

I'd recommend finding a different tutorial to help you get the Bluetooth module connected to your phone (sorry, but I haven't done any Windows phone development since we posted this Instructable). Or you could use a different method of controlling and visualizing the glove's functions (maybe using some hardware buttons and LEDs).

Makercool (author)2016-01-05

Nice project....... i loved it

my only question is that how can i make an android app for this. Please can you help me i'm in 9th standard and i have no clue about making an app.

if someone could explain it to me in detail ... it would be great.

Please reply fast

lstearns86 (author)Makercool2016-01-06

The bluetooth module and app aren't really necessary, they were just used to visualize the sensor readings and set the mode. You could just hardcode the mode, control it from your computer over USB, or hook up a button to cycle through the modes.

As for your other question, we did not use an uno, but really any small arduino compatible board would work.

Makercool (author)2016-01-05

i have used the bluetooth module hc06 will that be okay?

Makercool (author)2016-01-05

did u connect mini using arduino uno?

Makercool (author)2016-01-05

if someone could make the app it would a huge favour......

Đức BáchN (author)2015-10-12

Hey guys , Could you explain for me these lines belows (in void navloop)

int v = (readings[numFingers+i]-10)/150*256;

v=v>255 ? 255 : v;

v=v<0 ? 0 : v;

analogWrite(vibrations[i*3], 255-v);

lstearns86 (author)Đức BáchN2015-10-12

It's been awhile since I've worked on this code, but based on the comment above these lines it looks like this was taking the range readings from the two ultrasonic rangefinders (which are stored in the readings array after the IR reflectance sensors for each finger, hence the numFingers+i) and translating it into vibration intensity. We wanted the maximum vibration strength to be at 10cm and the minimum to be at 150cm, and the rest is just ensuring that the value we write to the motor is within the analog range of 0-255. Please let me know if you need further clarification.

Đức BáchN (author)lstearns862015-10-17

Thank you so much , guys .Surely, if i have another question , i will ask you . Thanks in advance :D

Essam Wageh (author)2015-09-14

Hey guys , this's actually great ,

but I really need to know , how the 'messages" and the "typing" modes are working !! because you didn't mention enough information about the both of them , so I need some explanation for these 2 modes , any help !!

lstearns86 (author)Essam Wageh2015-09-14

Hi, thanks for your interest in our project.

The massage and typing modes were a bit half-baked, just added to demonstrate some additional possibilities. You'd need to do some additional work to make them useful in the real world.

The massage (not messages) mode is simply a single command to turn on all of the vibration motors so that you can use the glove as a massage tool. This mode was added as a bit of a joke.

The typing mode uses the status of the 4 reflectance sensors as a binary input to generate ASCII text, but since there are only 4 bits of information you're obviously limited to 15 different characters. We basically sensed which fingers are down by measuring the reflectance value and then sent a character matching the up/down pattern over Bluetooth. See the Arduino code for the exact details.

Let me know if you have more questions.

Essam Wageh (author)lstearns862015-09-14

Thanks a lot for your answer , I'm gonna check the Arduino code out , actually I'm trying to make this project by myself , so sure if I have more questions I'm gonna ask you guys , and thanks in advance :) ...

YogeshwaraB (author)2015-09-02

guys how do i get the app?? i did all the connections and everything is working i just need the app now to change modes instantaneously. can i have the replay ASAP.. and great work guys

lstearns86 (author)YogeshwaraB2015-09-02

The code for the app is available on Github, and the link is at the end of the instructable.

mayank.saini.148 (author)2015-03-07

the simple sensor is available instead of IR Reflectance sensor QRE113 so can i use this in replacement of this

moreover i don't have same vibration motor i have a different one

plz tell me which properties to be the same while using this

plz reply fast

IwankuS (author)2015-01-18

hye im interesting in your project, em how can i get die block diagram for this project?

john1999 (author)2014-10-29

can anyone provide me the IC number

ansab (author)2014-10-28

Thank you very much for replying but I still did not find the app. All I can see are codes. Can you tell me the name and exact location of the app. Kindly can you please give us more information about the spp part, we did not understand that part.

lstearns86 (author)ansab2014-10-28

There is no stand alone app, just the code. In order to use it you will need a Microsoft developer account and Visual Studio installed to deploy the app. However, the Bluetooth control protocol is very simple (see the last step in the Instructable), so you could build your own app for another platform fairly easily.

ansab (author)2014-10-28

Please tell if we need to download any app or any code in the windows phone.

Please reply fast .I will be very thankful if you do

lstearns86 (author)ansab2014-10-28

Both the code for the arduino and the Windows Phone app are available on our github page linked to near the end of the instructable. Here's the link: https://github.com/lstearns86/handsight. The Windows Phone code is within the handsight folder.

The app is very simple, just a few buttons to control the arduino mode and some visual feedback for the values returned by the reflectance sensors. The only marginally complicated thing we did was set up the Bluetooth communication over the Bluetooth Serial Port Profile (SPP), which wasn't immediately obvious in the Windows Phone APIs at the time.

Let me know if you have any questions, although it's been quite awhile since I developed the app.

ansab (author)2014-10-27

where is the app for wimdows phone.......

plz answer fast......

ansab (author)2014-10-27

where is the app for wimdows phone.......

plz answer fast......

ansab (author)2014-10-27

where is the app for wimdows phone.......

plz answer fast......

marckivan (author)2014-08-11

Hello there!! you guys did an amazing job there by constructing that device. I'm just wondering if the device can sense drop-offs (like stairs or pitfalls)

Patuko (author)2014-05-02

Oh man, you guys rock so hard :D

I was wondering to build a project similar like this and i'm seeing really hard work here about it! :D Really nice.

Did you guys have iterate on it?

Will really want to know if you have been testing it on over this year and if you can "sense" images better.

Thank you so much!

lstearns86 (author)Patuko2014-05-02

Thank you for your comments and for your interest in our project!

We haven't iterated on this particular design (it was for a class project), but we are working on other related ideas. One thought we had was to increase the resolution of what the fingers could "see" by using a small camera like those found in optical mice or even ones used for medical purposes. There is a lot of room for innovation in this field, so I hope that you'll feel inspired to try your own designs.

Patuko (author)lstearns862014-05-03

That's a nice idea, like having analog camera feed converted to vibes(processed data to different vibes)? Or just changind de ir sensors to a laser sensor?

The awesomedary thing of this haptic devices is that I think that if a children learns to use it, maybe with years of practice they are able to actually "see" throught that.

In Spain theres a famous case of a daltonic person that can't see colors and uses a ir device that launches a freq. sound for each color in front of it (its also analog because he can "sense" the enviromnt general color when its measure at a distance)
The incredible thing of this is that he actually says that hes brain has begin(fourth year of use) to make colors appear!
This case went famous because he wanted to had that device implanted on his body to always had it on so noone could say to him to wear it off.
Thanks for answering :D

HighWing (author)2013-03-12

I've been thinking about making this myself and I thought I read somewhere that you had initally wanted to make it as a head mount but realized that the vibration motors on the head were a bad idea. Which is why you made them into a glove. However, I've used many glove devices for VR simulators and such, and I find that holding your hands out becomes tiresome pretty quickly.

So I was wondering about modifing your design to still put the dectors on a hat/headband, but move the motors to something you ware on a belt clip or somewhere else?


lstearns86 (author)HighWing2013-03-12

This is an interesting idea, and certainly feasible. Our original plan was actually to make a belt with the vibration motors sewn into the fabric, and the ultrasonic sensors placed at different angles around it. When we started focusing more on the tactile surface feedback via the reflectance sensors we decided it made more sense to put everything on the hand instead. I don't see any reason why you couldn't put the sensors and feedback mechanism in different places, especially if you're only interested in the navigation portion.

One of our early inspirations was the idea of a haptic compass like the one described here: http://www.gradman.com/hapticcompass. It uses multiple vibration motors to indicate direction, and can either point north or toward an arbitrary compass direction for guidance. It wouldn't be too hard to modify that idea to give feedback from the ultrasonic distance sensors instead (or as well).

If you want to put the sensors on the head, you could use long wires as long as their resistance isn't too high. An alternative would be to build a second arduino device and have them communicate over Bluetooth.

huntjulien (author)2012-12-18

crazy cool. you guys are amazing

About This Instructable

30,172views

124favorites

License:

Add instructable to: