Introduction: Accel Writing (Magic Hand)


The Magic Hand allows people with disabilities and motor skill impairments to enjoy the creativity of drawing and writing in a simulated environment. The Magic Hand is a wearable glove that senses the motion of your index finger and translates that into the drawing of lines on a computer screen.

Materials Needed

LSM9DOF Breakout Board --- $24.95 ---

Adafruit Feather with Wifi --- $18.95 ---

Female/Female Wires --- $1.95 ---

Tape/Velcro strips --- $3

Two magnets of equal strength --- Prices vary

How it works

By using an accelerometer, we can gather acceleration data for the y-axis which will help us determine when the user's finger is moving up and down. Due to the fact that our accelerometer measures acceleration with respect to the center of the earth we can't determine the acceleration of the x-axis (left or right). Luckily the LSM9DOF breakout board also contains a magnetometer which allows us to gather data on magnetic fields. We place two magnets 30 cm apart and have the glove in between. If the magnetic data read positive then we know the glove is moving right and vice versa. After all the data is collected in the accelerometer/magnetometer it sends the data via wire to the feather which is connected to a computer of wifi and then forwards the data to the computer which we can then use in our code.

Step 1: Physical Prototype 1

This prototype is meant to be glove sewn together loosely on the hand in order for it to slip over the electronic devices. The electronic device will then be attached by velcro to the under armor sleeve base combined with a basic glove on the hand. Then the green glove will slip over the base and the electronic devices....

Steps in making the prototype glove:

  • Get two pieces of fabric large enough to trace hand
  • Trace hand onto both pieces of fabric and cut them out
  • Put the two hand cut outs together so they are perfectly alligned
  • Next, to prepare the sewing machine, run the thread through the indicated spots on the machine
  • When the sewing machine is set up, lift the needle and place the two put-together pieces of fabric under the needle
  • Make sure the needle is lined up on the very edge of the fabric, start the machine, and sew along the edges of the fabric, while leaving the two pieces unsewn at the wrist so a hand can fit in.

Step 2: Physical Prototype 2

Our final prototype is a regular glove combined with Velcro strap that is adjustable to any wrist. The glove and strap are sewn together, and the electronic devices are attached to the glove via Velcro.

Steps in making the 2nd prototype of the glove:

  1. Purchase a glove, the material of the glove does not matter.
  2. Purchase a velcro wrist strap
  3. Purchase a portable battery
  4. Purchase Sticky Velcro
  5. With a sewing needle, attach the velcro wrist strap to the base of the glove
  6. The wrist strap should be able to adjust to different wrist sizes.
  7. Attach the sticky tape to the base of the accelerometer and attach it to the index finger of the glove
  8. Attach sticky tape to the feather and attach it to the top of the glove.
  9. Using wires connect the 3V3 pin in the feather to the VIN pin in the accelerometer
  10. Using wires connect the GND pin in the feather to the GND pin the accelerometer.
  11. Using wires connect the SCL pin in the feather to the SCL pin the accelerometer.
  12. Using wires connect the SDA pin in the feather to the SDA pin the accelerometer.
  13. Connect at-least a 5 volt battery through usb to the feather to provide power.

Step 3: Magnets

Step 1: Put the two magnets of equals strength across from each other.

Step 2: Measure out 30 cm gap between the two magnets

Step 3: place the Magnetometer exactly in the middle of the two magnets. You should receive data around 0 while its in the middle. If you receive a reading of zero skip to step 5.

Step 4: If the reading is not zero or close to zero then you must adjust the distance of the magnets. If the reading is negative move the left magnet a cm or 2 to left or until reading is zero. If it is positive do the same thing except with the right magnet.

Step 5: Write code that accepts the data from the magnetometer and reads if it positive or negative. If positive have the code draw a line to the right and if negative draw a line left.

Step 4: Code


In order to process data from the accelerometer, a client/server relationship must be established between the Adafruit feather and the server that processes the data (running on a laptop/desktop). Two code files will need to be created: one for the client (the Adafruit feather), and the other for the server (in this case, Jarod’s laptop). The client is written in C++, and the server is written in python. The language used for the client matters as Arduino is mainly a C++ language, and changing it to use a different language is difficult. The server can be written in any language, as long as it has network features.

Setting up the Client:

First, we will setup the client code. Most of the WiFi connection code is readily available through the Adafruit libraries. We begin by including relevant classes.

#include <ESP8266WiFi.h>
#include <SPI.h> #include <Wire.h> #include <Adafruit_Sensor.h> #include <Adafruit_LSM9DS0.h>

Set some variables what will be used throughout the code.

// Connect to a network
const char* ssid = "MMServer"; const char* password = "MMServer-Password"; // IP and port of the server which will receive data const char* host = ""; const int port = 12347; bool connected = false;

// Initialize motion detector Adafruit_LSM9DS0 lsm = Adafruit_LSM9DS0(1000);

WiFiClient client;

Create a setup() function which will be run as soon as the feather starts.

// Setup WiFi connection, and connect to the server
void setup() { Serial.begin(9600); delay(100);

Serial.println(); Serial.println(); Serial.print("Connecting to "); Serial.println(ssid); // Start WiFi WiFi.begin(ssid, password); // Connecting... while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } // Successfully connected to WiFi Serial.println(""); Serial.println("WiFi connected"); Serial.println("IP address: "); Serial.println(WiFi.localIP());

#ifndef ESP8266 while(!Serial); #endif Serial.begin(9600); Serial.println("Sensor Test");

// Initialize the sensor if(!lsm.begin()) { // There was a problem detecting the LSM9DS0 Serial.print(F("Ooops, no LSM9DS0 detected ... Check your wiring or I2C ADDR!")); while(1); } Serial.println(F("Found LSM9DS0 9DOF")); // Begin connecting to server Serial.print("Connecting to "); Serial.println(host);

// Check for successful connection. If failed then abort if (!client.connect(host, port)) { Serial.println("connection failed"); connected = false; return; } else { connected = true; }

//Setup the sensor gain and integration time configureSensor(); }

We then need a loop function that will repeatedly loop. In this case, it is used to repeatedly send data from the accelerometer to the server in the form of “[z_accel]:[y_mag]:[z_mag]”. The client.print(numbers); function is what sends data to the server.

void loop() {
delay(250); if(connected){ // This will send data to the server sensors_event_t accel, mag, gyro, temp; lsm.getEvent(&accel, &mag, &gyro, &temp); String numbers; numbers += accel.acceleration.z; numbers += ":"; numbers += mag.magnetic.y; numbers += ":"; numbers += mag.magnetic.z; Serial.print(numbers); client.print(numbers); Serial.println(); } else { establishConnection(); } }

For some utility functions, we need one to establish the connection between the feather and the server.

void establishConnection(){
if (!client.connect(host, port)) { Serial.println("connection failed"); connected = false; return; } else { connected = true; } }

We also need to configure the sensor and give it the range of values it will read. For example, acceleration has 5 options for the range: 2g, 4g, 6g, 8g, and 16g.

void configureSensor(void)
{ // Set the accelerometer range //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_2G); lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_4G); //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_6G); //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_8G); //lsm.setupAccel(lsm.LSM9DS0_ACCELRANGE_16G); // Set the magnetometer sensitivity //lsm.setupMag(lsm.LSM9DS0_MAGGAIN_2GAUSS); //lsm.setupMag(lsm.LSM9DS0_MAGGAIN_4GAUSS); //lsm.setupMag(lsm.LSM9DS0_MAGGAIN_8GAUSS); lsm.setupMag(lsm.LSM9DS0_MAGGAIN_12GAUSS);

// Setup the gyroscope lsm.setupGyro(lsm.LSM9DS0_GYROSCALE_245DPS); //lsm.setupGyro(lsm.LSM9DS0_GYROSCALE_500DPS); //lsm.setupGyro(lsm.LSM9DS0_GYROSCALE_2000DPS); }

Setting up the Server:

The server will be a python file that will run on the command line of a computer. To start, import the required classes.

import socket
import re import pyautogui

socket is used for networking. re is used for regex, or string manipulations. pyautogui is a python library which will allow the drawing to happen (discussed later).

Next, we should define some variables. These will be global variables, so they will be accessed in multiple functions. They will be used later in the code.

i = 0
n = 0 line = 1

data_list = []

mag_data = [] mag_calib_y = 0 mag_offset_y = 0

z_calib = 0 z_offset = 0 z_moving_offset = 0 z_diff = 0 z_real = 0 z_velo = 0 z_pos = 0

keep_offset = False first_data = True

We now need a function to create a server and open it for incoming connections.

def startServer():
global i global first_data # initialize server socket serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) # Server IP address and port host = "" port = 12347 server_address = (host, port) # Open the server and listen for incoming connections print ('Starting server on %s port %s' % server_address) serversocket.bind(server_address) serversocket.listen(5) # Wait for connections... while True: print ('Waiting for connection...') # Accept an incoming connection (clientsocket, address) = serversocket.accept() # Try to parse data received try: print ('Connection established from ', address) while True: # Receive the data and send it for processing data = clientsocket.recv(25) accel_data = re.split('[:]', str(data)) accel_data[0] = accel_data[0][2:] accel_data[1] = accel_data[1] accel_data[2] = accel_data[2][1:-1] print(accel_data) i+=1 if(i < 51): calibData(accel_data) else: movingAccel(accel_data[0]) processData(accel_data) first_data = False finally: # Close the socket to prevent unnecessary data leak clientsocket.close()

We now require the functions that will process all of the data. The first step to take, and the first function called, is the calibration of the sensor for the calculation purposes.

def calibData(list):
global z_calib global z_offset global mag_data global mag_calib_y global mag_offset_y z_calib += float(list[0]) mag_calib_y += float(list[1]) if(i==50): z_offset = z_calib / 50 mag_offset_y = mag_calib_y / 50 z_calib = 0 mag_calib_y = 0 mag_data.append(mag_offset_y)

Next, we create a moving acceleration offset. This makes it so the program recognizes when someone stops moving their finger because all of the values for acceleration that are sent to the server should be the same at that time.

def movingAccel(num):
global z_calib global z_diff global z_moving_offset global z_offset global data_list global n global keep_offset if(n < 10): n += 1 z_calib += float(num) data_list.append(float(num)) else: z_moving_offset = z_calib / 10 for entry in data_list: z_diff = float(entry) - z_moving_offset if(z_diff > 0.2 or z_diff < -0.2): # motion detected within data, restart keep_offset = True n = 0 z_calib = 0 z_moving_offset = 0 z_diff = 0 data_list = [] break if not keep_offset: # stationary in data, set new z_offset z_offset = z_moving_offset print("New z_offset: ") print(z_offset) n = 0 z_calib = 0 z_moving_offset = 0 z_diff = 0 data_list = [] keep_offset = False keep_offset = False

Next, we do the brunt of the math. This involves translating the acceleration data into a position data which will allow us to tell the direction that the user moves their finger.

def processData(list): #[accel.z, mag.y]
global z_offset global z_real global z_velo global z_pos global first_data global mag_data

z_real = float(list[0]) - z_offset mag_y = list[1] mag_z = list[2] left = False right = False # Don't process acceleration until absolutely sure it has accelerated # Prevents mechanical noise from contributing to position if(z_real < 0.20 and z_real > -0.20): z_real = 0 #Begin integrations to find position if(first_data): mag_data.append(mag_y) z_pos = (0.5 * z_real * 0.25 * 0.25) + (z_velo * 0.25) + z_pos z_velo = z_real * 0.25 pyautogui.moveTo(1500,1000) else: z_pos = (0.5 * z_real * 0.25 * 0.25) + (z_velo * 0.25) + z_pos z_velo = (z_real * 0.25) + z_velo del mag_data[0] mag_data.append(mag_y) if(float(mag_data[1]) - float(mag_data[0]) > 0.03): right = True elif(float(mag_data[1]) - float(mag_data[0]) < -0.03): left = True if(right): movement(50, int(z_pos*1000)) elif(left): movement(-50, int(z_pos*1000)) z_velo = 0 z_pos = 0

Now, finally, we move the cursor! To do this, we opened a paint window and made it full screen. The pyautogui library contains a function called pyautogui.dragRel(x,y); which we use to drag the mouse cursor from one point to the next. It uses relative position data so the movement is relative to the last position of the cursor.

def movement(x, y):
print("moving to", x, -y) pyautogui.dragRel(x,-y)

Lastly, we need to call the main function to even allow all of this code to run.

# Calls the function to begin the server