Introduction

The Magic Hand allows people with disabilities and motor skill impairments to enjoy the creativity of drawing and writing in a simulated environment. The Magic Hand is a wearable glove that senses the motion of your index finger and translates that into the drawing of lines on a computer screen.

Materials Needed

LSM9DOF Breakout Board --- $24.95 --- https://www.adafruit.com/product/2021

Adafruit Feather with Wifi --- $18.95 --- https://www.adafruit.com/product/3046

Female/Female Wires --- $1.95 --- https://www.adafruit.com/product/1950

Tape/Velcro strips --- $3

Two magnets of equal strength --- Prices vary

How it works

By using an accelerometer, we can gather acceleration data for the y-axis which will help us determine when the user's finger is moving up and down. Due to the fact that our accelerometer measures acceleration with respect to the center of the earth we can't determine the acceleration of the x-axis (left or right). Luckily the LSM9DOF breakout board also contains a magnetometer which allows us to gather data on magnetic fields. We place two magnets 30 cm apart and have the glove in between. If the magnetic data read positive then we know the glove is moving right and vice versa. After all the data is collected in the accelerometer/magnetometer it sends the data via wire to the feather which is connected to a computer of wifi and then forwards the data to the computer which we can then use in our code.

Project Proposal.pdf
Download

Step 1: Physical Prototype 1

This prototype is meant to be glove sewn together loosely on the hand in order for it to slip over the electronic devices. The electronic device will then be attached by velcro to the under armor sleeve base combined with a basic glove on the hand. Then the green glove will slip over the base and the electronic devices....

Steps in making the prototype glove:

Get two pieces of fabric large enough to trace hand
Trace hand onto both pieces of fabric and cut them out
Put the two hand cut outs together so they are perfectly alligned
Next, to prepare the sewing machine, run the thread through the indicated spots on the machine
When the sewing machine is set up, lift the needle and place the two put-together pieces of fabric under the needle
Make sure the needle is lined up on the very edge of the fabric, start the machine, and sew along the edges of the fabric, while leaving the two pieces unsewn at the wrist so a hand can fit in.

Step 2: Physical Prototype 2

Our final prototype is a regular glove combined with Velcro strap that is adjustable to any wrist. The glove and strap are sewn together, and the electronic devices are attached to the glove via Velcro.

Steps in making the 2nd prototype of the glove:

Purchase a glove, the material of the glove does not matter.
Purchase a velcro wrist strap
Purchase a portable battery
Purchase Sticky Velcro
With a sewing needle, attach the velcro wrist strap to the base of the glove
The wrist strap should be able to adjust to different wrist sizes.
Attach the sticky tape to the base of the accelerometer and attach it to the index finger of the glove
Attach sticky tape to the feather and attach it to the top of the glove.
Using wires connect the 3V3 pin in the feather to the VIN pin in the accelerometer
Using wires connect the GND pin in the feather to the GND pin the accelerometer.
Using wires connect the SCL pin in the feather to the SCL pin the accelerometer.
Using wires connect the SDA pin in the feather to the SDA pin the accelerometer.
Connect at-least a 5 volt battery through usb to the feather to provide power.

Step 3: Magnets

Step 1: Put the two magnets of equals strength across from each other.

Step 2: Measure out 30 cm gap between the two magnets

Step 3: place the Magnetometer exactly in the middle of the two magnets. You should receive data around 0 while its in the middle. If you receive a reading of zero skip to step 5.

Step 4: If the reading is not zero or close to zero then you must adjust the distance of the magnets. If the reading is negative move the left magnet a cm or 2 to left or until reading is zero. If it is positive do the same thing except with the right magnet.

Step 5: Write code that accepts the data from the magnetometer and reads if it positive or negative. If positive have the code draw a line to the right and if negative draw a line left.

Step 4: Code

https://github.iu.edu/ise-e101-F17/MuscleMemory-Sw...

Introduction:

In order to process data from the accelerometer, a client/server relationship must be established between the Adafruit feather and the server that processes the data (running on a laptop/desktop). Two code files will need to be created: one for the client (the Adafruit feather), and the other for the server (in this case, Jarod’s laptop). The client is written in C++, and the server is written in python. The language used for the client matters as Arduino is mainly a C++ language, and changing it to use a different language is difficult. The server can be written in any language, as long as it has network features.

Setting up the Client:

First, we will setup the client code. Most of the WiFi connection code is readily available through the Adafruit libraries. We begin by including relevant classes.

#include <ESP8266WiFi.h>
#include <SPI.h>
#include <Wire.h>
#include <Adafruit_Sensor.h>
#include <Adafruit_LSM9DS0.h>

Set some variables what will be used throughout the code.

// Connect to a network 
const char* ssid     = "MMServer";
const char* password = "MMServer-Password";
 
// IP and port of the server which will receive data
const char* host = "149.160.251.3";
const int port = 12347;
bool connected = false;
// Initialize motion detector
Adafruit_LSM9DS0 lsm = Adafruit_LSM9DS0(1000);
WiFiClient client;

Create a setup() function which will be run as soon as the feather starts.

// Setup WiFi connection, and connect to the server
void setup() {
  Serial.begin(9600);
  delay(100);
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
  
  // Start WiFi
  WiFi.begin(ssid, password);
  
  // Connecting...
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  
  // Successfully connected to WiFi
  Serial.println("");
  Serial.println("WiFi connected");  
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  #ifndef ESP8266
    while(!Serial);
  #endif
    Serial.begin(9600);
    Serial.println("Sensor Test");
  // Initialize the sensor
  if(!lsm.begin())
  {
    // There was a problem detecting the LSM9DS0
    Serial.print(F("Ooops, no LSM9DS0 detected ... Check your wiring or I2C ADDR!"));
    while(1);
  }
  Serial.println(F("Found LSM9DS0 9DOF"));
  
  // Begin connecting to server
  Serial.print("Connecting to ");
  Serial.println(host);
  // Check for successful connection. If failed then abort
  if (!client.connect(host, port)) {
    Serial.println("connection failed");
    connected = false;
    return;
  } else {
    connected = true;
  }
  //Setup the sensor gain and integration time
  configureSensor();
}

We then need a loop function that will repeatedly loop. In this case, it is used to repeatedly send data from the accelerometer to the server in the form of “[z_accel]:[y_mag]:[z_mag]”. The client.print(numbers); function is what sends data to the server.

void loop() {
  delay(250);
  if(connected){
    // This will send data to the server
    sensors_event_t accel, mag, gyro, temp;
    lsm.getEvent(&accel, &mag, &gyro, &temp);
  
    String numbers;
    numbers += accel.acceleration.z;
    numbers += ":";
    numbers += mag.magnetic.y;
    numbers += ":";
    numbers += mag.magnetic.z;
    Serial.print(numbers);
    client.print(numbers);
    Serial.println();
  } else {
    establishConnection();
  }
}

For some utility functions, we need one to establish the connection between the feather and the server.

void establishConnection(){
  if (!client.connect(host, port)) {
    Serial.println("connection failed");
    connected = false;
    return;
  } else {
    connected = true;
  }
}

We also need to configure the sensor and give it the range of values it will read. For example, acceleration has 5 options for the range: 2g, 4g, 6g, 8g, and 16g.

void configureSensor(void)
{
  // Set the accelerometer range
  //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_2G);
  lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_4G);
  //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_6G);
  //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_8G);
  //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_16G);
  
  // Set the magnetometer sensitivity
  //lsm.setupMag(lsm.LSM9DS0_MAGGAIN_2GAUSS);
  //lsm.setupMag(lsm.LSM9DS0_MAGGAIN_4GAUSS);
  //lsm.setupMag(lsm.LSM9DS0_MAGGAIN_8GAUSS);
  lsm.setupMag(lsm.LSM9DS0_MAGGAIN_12GAUSS);
  // Setup the gyroscope
  lsm.setupGyro(lsm.LSM9DS0_GYROSCALE_245DPS);
  //lsm.setupGyro(lsm.LSM9DS0_GYROSCALE_500DPS);
  //lsm.setupGyro(lsm.LSM9DS0_GYROSCALE_2000DPS);
}

Setting up the Server:

The server will be a python file that will run on the command line of a computer. To start, import the required classes.

import socket
import re
import pyautogui

socket is used for networking. re is used for regex, or string manipulations. pyautogui is a python library which will allow the drawing to happen (discussed later).

Next, we should define some variables. These will be global variables, so they will be accessed in multiple functions. They will be used later in the code.

i = 0
n = 0
line = 1
data_list = []
mag_data = []
mag_calib_y = 0
mag_offset_y = 0
z_calib = 0
z_offset = 0
z_moving_offset = 0
z_diff = 0
z_real = 0
z_velo = 0
z_pos = 0
keep_offset = False
first_data = True

We now need a function to create a server and open it for incoming connections.

def startServer():
	global i
	global first_data
	# initialize server socket
	serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
	serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
	
	# Server IP address and port
	host = "149.160.251.3"
	port = 12347
	server_address = (host, port)
	
	# Open the server and listen for incoming connections
	print ('Starting server on %s port %s' % server_address)
	serversocket.bind(server_address)
	serversocket.listen(5)
	
	# Wait for connections...
	while True:
		print ('Waiting for connection...')
		
		# Accept an incoming connection
		(clientsocket, address) = serversocket.accept()
		
		# Try to parse data received
		try:
			print ('Connection established from ', address)
			
			while True:
				# Receive the data and send it for processing
				data = clientsocket.recv(25)
				accel_data = re.split('[:]', str(data))	
				accel_data[0] = accel_data[0][2:]
				accel_data[1] = accel_data[1]
				accel_data[2] = accel_data[2][1:-1]
				print(accel_data)
				i+=1
				if(i < 51):
					calibData(accel_data)
				else:
					movingAccel(accel_data[0])
					processData(accel_data)
					first_data = False
		finally:
			# Close the socket to prevent unnecessary data leak
			clientsocket.close()

We now require the functions that will process all of the data. The first step to take, and the first function called, is the calibration of the sensor for the calculation purposes.

def calibData(list):
	global z_calib
	global z_offset
	global mag_data
	global mag_calib_y
	global mag_offset_y
	
	z_calib += float(list[0])
	mag_calib_y += float(list[1])
	
	if(i==50):
		z_offset = z_calib / 50
		mag_offset_y = mag_calib_y / 50
		z_calib = 0
		mag_calib_y = 0
		mag_data.append(mag_offset_y)

Next, we create a moving acceleration offset. This makes it so the program recognizes when someone stops moving their finger because all of the values for acceleration that are sent to the server should be the same at that time.

def movingAccel(num):
	global z_calib
	global z_diff
	global z_moving_offset
	global z_offset
	global data_list
	global n
	global keep_offset
	
	if(n < 10):
		n += 1
		z_calib += float(num)
		data_list.append(float(num))
	else:
		z_moving_offset = z_calib / 10
		for entry in data_list:
			z_diff = float(entry) - z_moving_offset
			if(z_diff > 0.2 or z_diff < -0.2): # motion detected within data, restart
				keep_offset = True
				n = 0
				z_calib = 0
				z_moving_offset = 0
				z_diff = 0
				data_list = []
				break
		if not keep_offset: # stationary in data, set new z_offset
			z_offset = z_moving_offset
			print("New z_offset: ")
			print(z_offset)
			n = 0
			z_calib = 0
			z_moving_offset = 0
			z_diff = 0
			data_list = []
			keep_offset = False
		keep_offset = False

Next, we do the brunt of the math. This involves translating the acceleration data into a position data which will allow us to tell the direction that the user moves their finger.

def processData(list): #[accel.z, mag.y]
	global z_offset
	global z_real
	global z_velo
	global z_pos
	global first_data
	global mag_data
	z_real = float(list[0]) - z_offset
	mag_y = list[1]
	mag_z = list[2]
	left = False
	right = False
	
	# Don't process acceleration until absolutely sure it has accelerated
	# Prevents mechanical noise from contributing to position
	if(z_real < 0.20 and z_real > -0.20):
		z_real = 0
	
	#Begin integrations to find position
	if(first_data):
		mag_data.append(mag_y)
		z_pos = (0.5 * z_real * 0.25 * 0.25) + (z_velo * 0.25) + z_pos
		z_velo = z_real * 0.25
		pyautogui.moveTo(1500,1000)
	else:
		z_pos = (0.5 * z_real * 0.25 * 0.25) + (z_velo * 0.25) + z_pos
		z_velo = (z_real * 0.25) + z_velo
		
		del mag_data[0]
		mag_data.append(mag_y)
		if(float(mag_data[1]) - float(mag_data[0]) > 0.03):
			right = True
		elif(float(mag_data[1]) - float(mag_data[0]) < -0.03):
			left = True
		
		if(right):
			movement(50, int(z_pos*1000))
		elif(left):
			movement(-50, int(z_pos*1000))
		z_velo = 0
		z_pos = 0

Now, finally, we move the cursor! To do this, we opened a paint window and made it full screen. The pyautogui library contains a function called pyautogui.dragRel(x,y); which we use to drag the mouse cursor from one point to the next. It uses relative position data so the movement is relative to the last position of the cursor.

def movement(x, y):
	print("moving to", x, -y)
	pyautogui.dragRel(x,-y)

Lastly, we need to call the main function to even allow all of this code to run.

# Calls the function to begin the server
startServer()

Accel Writing (Magic Hand)

Introduction

Materials Needed

How it works

Step 1: Physical Prototype 1

Step 2: Physical Prototype 2

Step 3: Magnets

Step 4: Code

Introduction:

Setting up the Client:

Setting up the Server:

Recommendations

Introduction: Accel Writing (Magic Hand)

Introduction

Materials Needed

How it works

Attachments

Step 1: Physical Prototype 1

Step 2: Physical Prototype 2

Step 3: Magnets

Step 4: Code

Introduction:

Setting up the Client:

Setting up the Server:

Recommendations

Books and Bookshelves Contest

Big and Small Contest

Making Time Contest