Making Tactile 3D Prints Talk

The Arduino Duemilanove microcontroller was used in the instructables named “Capacitive-Touch Arduino Keyboard Piano” to demonstrate how capacitance sensing can be used to simulate playing a piano by touching the aluminum foil “keys” (Nicholas Jones and Tyler Crumpton, https://www.instructables.com/id/Capacitive-Touch-Arduino-Keyboard-Piano/). This instructable led me to hire two students from the University of Texas at El Paso College of Engineering (Cesar Valle and Luis Banuelos) who helped me re-work the C++ Arduino code from the piano instructables so that the code now worked with that provided for the Spikenzie VoiceShield which permits audio clips to be stored and accessed using the modified version of the Arduino code. This provided a way to use capacitance sensing to generate an audio response when a finger touched a 3D print of a microscope slide image so that the audio response now can tell the learner what they are touching on the 3D print. In April, 2014, I published a paper titled “3D Printed Tactile Learning Objects: Proof of Concept” in the Journal of Blindness Innovation and Review (see https://nfb.org/images/nfb/publications/jbir/jbir14/jbir040102.html) showing how the software PhotoToMesh can make 3D prints from 2D photos taken when using a microscope or telescope. I will show in this instructable how a 3D print of a cell in the anaphase stage of mitosis can be provided with a capacitance sensing microcontroller option for generating audio telling the learner the name of the structure being touched on the 3D print. This is especially useful for blind or visually impaired students who can now hear an audio response telling them where they are touching the 3D print so they can more easily learn what sighted students see when viewing a slide with a microscope. This prototype for making microscope slides more accessible for blind or visually impaired students would also work just as well with astronomy 3D prints made from published 2D photos taken with the Hubble telescope.

Step 1: Making 3D Print Files of Microscope Slide Images:

What is needed:

PhotoToMesh (Ransen Software, http://www.ransen.com/phototomesh/)

Computer with Adobe Photoshop

Image file in jpeg format

meshlab software (free for Mac, use to check stl file)

netfabb software (use free version for minor stl file cuts)

Making an stl file with PhotoToMesh:

If you wish to use your own jpeg image file, you will have to adjust the color pattern using a program like Adobe Photoshop. PhotoToMesh raises white or lighter colors higher on the z axis and lowers black or darker colors on the z axis of the produced stl file needed for 3D printing. The graphic shows an example of how an original image on the left can be made black and white and then color inverted to produce an image (middle image) where the chromosomes and cell membrane, for example, are white so that the 3D print produced from this file will have the chromosomes and cell membrane raised higher than that of the surrounding area. The goal, of course, is to produce a tactile image that can by touch and feel provide information for a blind student to learn where the chromosomes and cell membrane are located. The image at far right shows the 3D print made from an stl file by PhotoToMesh starting with the central image.

The stl file produced by PhotoToMesh can be viewed as a 3D image before 3D printing by using meshlab, a free software that can be downloaded at http://meshlab.sourceforge.net/. If some simple cuts need to be made to the stl file, another free software named netfabb Basic can be used and a reduced function version can be downloaded for no charge at http://www.netfabb.com/basic.php.

Step 2: Where Can You Get Already Made 3D Print Files of 2D Images:

If you wish to use the same stl file used in this instructable, go to the NIH 3D Print Exchange at http://3dprint.nih.gov/discover/3dpx-000519 to download the file as a zip file by clicking on the downward pointing arrow on the bottom right of the 3D print image as indicated in the above graphic. I have uploaded several stl files for mitosis to this site as well as for other tissues and any of these could be used for making a 3D print.

Make a 3D print on your own if you have access to a 3D printer either at home or at a local MakerSpace. If you wish to have an outside 3D printing company make your 3D print, I have had good luck with i.Materialize at http://i.materialise.com/3dprintlab/instance/111c... where a 3D print can be made for around $17.00 a print for the size used in this instructable.

Step 3: Microcontroller:

What is needed:

Arduino Duemilanove

Spikenzie VoiceShield

Arduino Duemilanove can be purchased from many places but one source at http://www.best-studio-headphones.com/product_inf... is selling now for $23.99. The Spikenzie VoiceShield can be purchased fully assembled for $55.95 at http://www.best-studio-headphones.com/product_inf...

The VoiceShield fits on top of the Duemilanove and uses digital pins 2, 3, 4 and 5 of the Arduino leaving available pins 6 – 12 for this project. The VoiceShield “voice chip” can hold 4 minutes of audio but for this project, the software for loading my own speech clips was set for 3 seconds per clip which means that up to 80 audio clips can be stored on the “voice chip”. (240 seconds/80 = 3 seconds per audio clip). A basic introduction to the VoiceShield can be seen at http://www.spikenzielabs.com/SpikenzieLabs/VoiceS...

Before loading your own audio onto the “voice chip”, the VoiceShield (VS) library must be installed along with the VS Loader sketch which is the software that programs the “voice chip” with your own audio clips. Go to http://www.spikenzielabs.com/SpikenzieLabs/VoiceS... for instructions showing how to load the VS Library and VS programming software onto your computer so that it is set for loading your own audio clips onto the “voice chip”.

Step 4: Preparing to Load Audio Files Onto “voice Chip”:

Your own audio clips can be in many audio file formats but for this study, I used the mp3 format. I also used GarageBand on my MacBook Pro to record my audio and edited the audio clips to be below 3 seconds in length. I further shortened the audio clips in Final Cut Pro to fine-tune the clips to as short as needed to produce the desired audio clip with little blank recording space. The VS Programmer software is required for loading your own audio onto the “voice chip” and this can be downloaded for your type of computer from http://www.spikenzielabs.com/SpikenzieLabs/VoiceS....

The VS Programmer software looks for two files. One contains the audio clips you wish to load onto the “voice chip” and they must be stored in a folder named “SoundBytes”. For this instructable, I used 80 audio clips beginning with a zero and ending with 79. See the image showing a portion of SoundScore.txt with 0 – 29 audio clips. The number to the left of an audio file is requested when that file is to be played. When making the text file, each text file line begins with a number, then a tab followed next by the name of the audio file followed by a return key to begin the next entry. The last entry of an audio file must also be followed by a return key entry from your keyboard. The SoundBytes folder with audio files must also be in the same folder as the SoundScore.txt file.

Step 5: Loading Audio Files Onto the Voice Chip:

Once the SoundScore text file and SoundBytes folder are completed, you can download the VSProgrammerLite software (http://www.spikenzielabs.com/SpikenzieLabs/VoiceShield_Software.html) for your type of computer. Running that software shows a screen as shown above

Select the Port you are using to connect your computer to the Arduino-VoiceShield by entering the port number. Once that is done, select “Program” from the top list and you should now see each audio file being loaded by seeing the name of the file being loaded shown in the Status section at the bottom of the VSProgrammerLite window. You can also look below the VoiceShield to see an LED on the Arduino turning on and off as each audio file is loaded onto the VoiceShield chip.

.

Step 6: Checking VoiceShield for Successful Audio Loading:

I’ve attached a set of speakers to the Arduino so once the audio clips have been loaded onto the VoiceShield, it is good to run a test to search for files using the sketch listed at http://www.spikenzielabs.com/SpikenzieLabs/VoiceS... and shown above.

Substitute any number from zero to 79 in the line – vs.ISDPLAY_to_EOM(5) to replace “5”, save the file and reload sketch onto the VoiceShield. The audio clip played will be what is indicated from the text file for that number and will replay at 2 second intervals.

Step 7: From Breadboard to Arduino:

What is needed:

breadboard

six 2 MegOhm resistors

18 connecting wires

Note that the first 2 MegOhm resistor connects to Pin 6 and serves as the reference capacitance reading to compare with each of the other connections to pins 7 – 12. Each of the green wires connects to a capacitance sensor composed of a thin copper foil or in one case to the bare wire in capacitance sensor 6. When the sketch is loaded to test for touch leading to an audio response, just touching a green wire with your fingers will cause a change in capacitance to generate an audio response for that particular green wire and provides an easy way to test the breadboard for correct wire connections.

Step 8: Making Capacitance (cap) Sensors for Cap Sensor Layer:

What is needed:

six connecting wires

2D print of original 3D print file

~6 floor protective pads

1 poster board

When two plates in a circuit are placed close together, the charge that exists between these two plates is called the capacitance. In this instructable, a copper plate in the circuit has a capacitance that can be changed by a finger touching it. A finger carries a charge resulting from the electrical activity required to cause normal heart contraction so that when a finger touches a copper plate, the capacitance is changed and can be detected by the sketch software when it changes above the set threshold. The interesting discovery is that a change in capacitance can be detected even though the copper plate is on one side of the 3D print and the finger is on the other side. A 3D print made from either polylactic acid (PLA filament) or polyamide permits capacitance to be changed using the cap sensing mechanism employed in this instructable. The plastic 3D prints do not conduct electricity so that it must be the field set up by the capacitance on the copper plate that is being altered by a finger touch.

In the above image, capacitance (cap) sensors 1 – 4 are composed of copper foil place on a cut-to-fit floor protective pad that was found to have a sticky material that held the copper foil tightly. Cap sensor 5 was found to work well with just the bare wire attached to the sticky side of the floor protective pad. Cap sensors 1 – 5 are held in place on the 2D print by using double-sided tape. The 2D print in color was taped onto a slightly larger poster board and the 2D print served as a useful way to know where to attach each of cap sensors 1 – 5 so they would be underneath the same structural area in the 3D print that is going to be placed on top of the cap sensor layer. Cap sensor 6 is attached to a piece of copper foil underneath a Braille 3D print spelling out the word anaphase so that a student can read the Braille for anaphase and also hear the cap sensor generate the audio response for anaphase.

Step 9: Checking Capacitance Sensor Layer Without 3D Print:

What is needed:

Cap sensor layer

By checking each cap sensor at this stage, you verify that each cap sensor is working properly before adding the top 3D print layer.

Step 10: Preparation of 3D Print to Place on Top of Cap Sensing Layer:

What is needed:

3D print of anaphase stage of mitosis

3D print of Anaphase in Braille

4 furniture protective pads

The furniture protective pads stick very well to the underside of the polyamide or polylactate plastic making up a 3D print. The placement of the protective pads and their height above the sensor layer make it an easy fit to turn over the 3D print and set it on top of the sensor layer. The cap sensing “sandwich” made up of the cap sensor layer with the 3D print on top provides a useful tool for learning about the important features that determine the stage of mitosis that this cell is in.

Step 11: Tactile 3D Print With Audio Response

The anaphase stage of mitosis can now be studied by both touching the surface of the 3D print and also by hearing the audio response giving the bumps and shapes names useful in determining the stage of mitosis that this cell is passing through. Relying on a tactile method for the blind student has been the traditional way to learn science but introducing audio to reinforce touch makes the learning process more complete. This is the first prototype using 3D printed images linked to touch and sound and portends a future in which many more images from microscopes and telescopes will be designed for students who are blind or visually impaired.

Step 12: Sketch Software for Six Cap Sensors:

#include <CapacitiveSensor.h>

#include<VoiceShield.h>

/*

* Touch Event Sound Player

*

* Using the SpikenzieLab's Voice Shield, play a different sound depending

* on which touch sensor was pressed.

*

* Version 0.1 - October 12, 2014

* Luis Carlos Bañuelos Chacon and Cesar Valle

* [e-mail]

*/

/*

* Instructions

*

* 1. On configuration,

* modify CS_NUM with the number of sensors needed,

* modify CS_THRESHOLD by decreasing the number if the sensors don't respond

* and increasing the number if they are too sensitive,

* modify VS_VOICES to match the voices programmed in the Voice Shield.

*

* 2. On array of capacitive sensors, initialize the touch sensors by indicating the

* common pin and the pin in which the sensor is connected.

*

* 3. On array of sounds, in the same order as the array of capacitive sensors indicate

* the ID of the sound to be played when the sensor is pressed.

*/

/*

* Configuration

*/

#define CS_NUM 6 // How many touch sensors

#define CS_SAMPLES 30 // How many samples to take when measuring the cap sensor

#define CS_THRESHOLD 30 // Reading threshold to consider as a touch event

#define VS_VOICES 80 // Number of voices programmed on voice shield

/*

* Array of capacitive sensors

*

* First parameter is the common pin, it should be the same for all instances.

* Second parameter is the sensor pin.

*/

CapacitiveSensor cs[] = {

CapacitiveSensor(6, 7),

CapacitiveSensor(6, 8),

CapacitiveSensor(6, 9),

CapacitiveSensor(6, 10),

CapacitiveSensor(6, 11),

CapacitiveSensor(6, 12),

};

/*

* Array of sounds

*

* This array holds the ID of the sound each sensor has to make.

* The order should correspond to the array of capacitive sensors.

*/

int sound[] = {

13,

28,

38,

29,

19,

4

};

/*

* Instantiate Voice Shield

*/

VoiceShield vs(VS_VOICES);

/*

* readCS

*

* Reads the array of capacitive sensors and returns

* which sensor is touched or -1 if none.

*/

int readCS() {

for (int i = 0; i < CS_NUM; i++) {

if (cs[i].capacitiveSensor(CS_SAMPLES) > CS_THRESHOLD) return i;

}

return -1;

}

void setup() {vs

;//pinMode(13, OUTPUT);

}

void loop() {

int cs_pressed = readCS(); // Read array of sensors

if (cs_pressed >= 0) { // If a sensor is touched

//digitalWrite(13, 1); // Indicate event on LED

vs.ISDPLAY_to_EOM(sound[cs_pressed]); // Play sound

//delay(1000); // Small delay for LED

//digitalWrite(13, 0); // Turn off led

//while(readCS() == cs_pressed); // Block until sensor is released

}

}