Storytelling Interface Pt 3: Interactive Music Poster

Music producers use many types of electronic devices to make sounds including drum machines and something called a MIDI controller (MIDI stands for Musical Instrument Digital Interface), which is a special type of keyboard that transmits and receives midi information or data. In this example musician Karriem Riggins talks about the influence of the MIDI controller (i.e. via J Dilla) on his drumming.

The interactive music poster project is a new addition to the storytelling interface series. Students can make a poster with images and each image is a sensor that, when touched, triggers audio. This project is inspired by the interactive paintings of Thomas "Detour" Evans. His portraits of iconic figures are hooked up to multiple Touch Boards devices and can be played like a MIDI controller.

MIDI controllers and other devices were used by legendary producers such as the late James Dewitt Yancey, better known as J Dilla (see video above), who collaborated with many hip-hop performers—from Questlove to Erykah Badu to Eminem. Dilla's MIDI Production Center 3000 is currently on view at the Smithsonian.

For this project, students will use poster board, markers and/or printed images, conductive paint and Touch Boards to make their own interactive music poster.

Supplies

Poster board

Card stock (ex. index cards)

Markers (pencils optional)

Masking tape

Scissors

Paintbrushes and cups

Alligator or crocodile clips

Conductive paint (i.e., Electric Paint)

Touch Board (up to 12 audio tracks)

Step 1: Explore Science & Engineering Contexts

Chemistry emphasizes science and engineering practices related to design and evaluation as well as investigation and modeling. A major focus of chemistry is on matter and its interactions.

Using conductive paint is a great way to explore these concepts because it contains graphite, or other materials that conduct electricity. Graphite is a crystalline compound and it is used to produce artificial diamonds, an allotropic modification of carbon.

Graphite is a very good conductor of electricity due to the mobility of electrons across layers of atoms to generate electricity. Molecules of graphite or other solid materials is present/active in conductive paint. This means that, when dry, the paint can act like a capacitive proximity sensor (or see this link), and trigger events such as playing sounds when touched.

Now, imagine creating and arranging several of these painted sensors to create a MIDI music making device.

Step 2: Find or Create Audio Content

Students will need to find or create their own audio content for the project. If the rights are cleared, they can use existing music (preferably with lyrics) or record their own sounds. Each conductive paint sensor will trigger a different sound file, so they may need to split up a larger/longer audio file.

Students can use Reverb to record their audio. Also, there are several audio recorders available but some are less expensive than others such as this one. Students can also use online tools to record audio, such as in Scratch. They can record and save up to 12 separate sound files (per device) as MP3s only.

Note: Try to keep each audio (MP3) file under 30 seconds. The shorter the file the less time it takes to load/play.

Step 3: Create the Music Visuals and Poster

Students need to find or create images based on each audio file or track. For example, a song can be used and images that illustrate the lyrics can be found online (from stock libraries), then printed out. Students can also draw and illustrate the audio tracks using pencil/marker and paper.

Next, cut out and paste the images on the poster board. Make sure the images don't overlap because each will become an electronic sensor that when touched triggers an audio track/sound.

Step 4: Create the Electronic Sensors

Using the conductive paint, create sensors or patches that are around the same size of each image (attached to the front). These sensor/patches, when dry, will be taped behind each image, on the back of the poster. Students can use a marker to write a number on the back of each patch to distinguish it from the others, especially if the audio playback needs to be sequential (ex. story, poem, song).

Make sure to leave gaps in the tape to attach alligator clips (clips must make contact with the painted areas). Tape the clips and connector wires to the back of the poster and make sure none of them touch or overlap.

Step 5: Prepare the Touch Board

Each Touch Board comes pre-packaged with a micro SD card in a slot on the device (see photo). This card already has sound files saved on it that can be replaced with new ones. To replaced the files, remove the micro SD card and insert it in a card reader.

Make sure that each sound file is saved as an MP3. If they aren't in MP3 format, you can always use an online MP3 converter.

Each Touch Board follows a naming structure: TRACK000.mp3 is for electrode E0, TRACK001.mp3 for E1, and so on, up to TRACK011.mp3 for E11. You'll see "E0" through "E11" marked on the edge of your Touch Board near the gold-plated electrodes.

Note: You do not need to re-program the Touch Board.

Step 6: Connect the Clips

Use alligator/crocodile clips to connect the painted electronic sensors to the Touch Board. Each sensor needs a separate clip; one end clips on to the patch and the other to an electrode on the Touch Board.

It is recommended that you secure the Touch Board on the back of the poster but it's not necessary.

In addition to alligator clips you can use other materials: see here.

Step 7: Try It Out

Once all of the alligator/crocodile clips are connected to the Touch Board electrodes plug the device into a power source (computer or outlet), then try touching the images on the poster. You should experience something like this:

Once the students create the posters they may want to create something more permanent such as an interactive wall mural or even an outdoor installation. They can "cold solder" the MIDI bridge on the Touch Board and turn other objects into MIDI controllers. Cold soldering refers to the use of conductive paint to connect the two sides of the bridge. Students can also connect the Touch Board to other software and use it as a MIDI controller. In this example, a student reprogrammed the Touch Board to trigger video clips instead of sounds:

In another example, students reprogrammed a Touch Board as a theremin, which is an instrument that is controlled without physical contact (i.e., no touching):

Using multiple Touch Boards and/or capacitive sensors, students can create their own interactive art installations that provide others with a variety of experiences (sight, sound, touch).