Ultrasonic Pi Piano With Gesture Controls!

by andygrove

9.8K4825

Intro: Ultrasonic Pi Piano With Gesture Controls!

This project uses inexpensive HC-SR04 ultrasonic sensors as inputs and generates MIDI notes that can be played through a synthesizer on the Raspberry Pi for a high quality sound.

The project also uses a basic form of gesture control, where the musical instrument can be changed by holding your hands over the two outermost sensors for a few seconds. Another gesture can be used to shut down the Raspberry Pi once you are finished.

The video above shows the finished product in a 3D-printed enclosure. There is a more in-depth video later in this instructable that explains how the project works.

Please also check out the latest documentation and tutorials here, which now includes information on a Python version of this project (this instructable uses the Rust programming language).

You can also buy the Ultrasonic Pi Piano as a kit here.

STEP 1: Ingredients

For this instructable, you will need the following ingredients:

  • Raspberry Pi (2 or later) with SD card
  • 8 HC-SR04 ultrasonic sensors
  • Octasonic Breakout Board
  • Bi-Directional Logic Level Converter
  • 8 x 4-pin molex jumper wires (female-female) for connecting the ultrasonic sensors to the Octasonic breakout
  • 13 x 6" Female-Female Jumper Wires for connecting the Raspberry Pi, Octasonic, and Logic Level Converter
  • Suitable power supply for the Raspberry Pi
  • PC speakers or similar

I would recommend using a Raspberry Pi 3 if possible since it has more computing power, resulting in a more responsive and pleasing sound. It can work fine with a Raspberry Pi 2 with a little tweaking but I would not attempt to use the original Raspberry Pi for this project.

HC-SR04 ultrasonic sensors have 4 connections - 5V, GND, Trigger, and Echo. Typically, Trigger and Echo are connected to separate pins on a microcontroller or Raspberry Pi but that means you would need to utilize 16 pins to connect 8 sensors, and this is not practical. This is where the Octasonic breakout board comes in. This board connects to all of the sensors and has a dedicated microcontroller that monitors the sensors and then communicates with the Raspberry Pi over SPI.

HC-SR04 require 5V and the Raspberry Pi is only 3.3V, so this is why we also need the logic level converter that will connect the Raspberry Pi to the Octasonic breakout board.

STEP 2: Connect the Ultrasonic Sensors to the Octasonic Board

Use 4 female-female jumper wires to connect each ultrasonic sensor to the board, being careful to connect them the correct way around. The board is designed so that the pins are in the same order as the pins on the ultrasonic sensor. From left to right on the board, the pins are GND, Trigger, Echo, 5V.

STEP 3: Connect the Logic Level Converter to the Octasonic Board

The Raspberry Pi and Octasonic Board communicate over SPI. SPI uses 4 wires:

  • Master In, Slave Out (MISO)
  • Master Out, Slave In (MOSI)
  • Serial Clock (SCK)
  • Slave Select (SS)

Additionally, we need to connect power (5V and GND).

The logic level converter has two sides - a low voltage (LV) and a high voltage (HV). The Raspberry will connect to the LV side since it is 3.3V. The Octasonic will connect to the HV side since it is 5V.

This step is for connecting the Octasonic to the HV side of the logic level converter.

See the photo attached to this step showing which pins should be connected to the logic level converter.

The connections from the Octasonic to the Logic Level converter should be as follows:

  • 5V to HV
  • SCK to HV4
  • MISO to HV3
  • MOSI to HV2
  • SS to HV1
  • GND to GND

STEP 4: Connect the Logic Level Converter to the Raspberry Pi

The Raspberry Pi and Octasonic Board communicate over SPI. SPI uses 4 wires:

  • Master In, Slave Out (MISO)
  • Master Out, Slave In (MOSI)
  • Serial Clock (SCK)
  • Slave Select (SS)

Additionally, we need to connect power (3.3V and GND). The logic level converter has two sides - a low voltage (LV) and a high voltage (HV). The Raspberry will connect to the LV side since it is 3.3V. The Octasonic will connect to the HV side since it is 5V.

This step is for connecting the Raspberry Pi to the LV side of the logic level converter.

The connections from the Raspbery Pi to the Logic Level converter should be as follows:

  • 3.3V to LV
  • GPIO11 (SPI_SCLK) to LV4
  • GPIO09 (SPI_MISO) to LV3
  • GPIO10 (SPI_MOSI) to LV2
  • GPIO08 (SPI_CE0_N) SS to LV1
  • GND to GND

Use the diagram attached to this step to locate the correct pins on the Raspberry Pi!

STEP 5: Connect Raspberry Pi 5V to Octasonic 5V

There is one final wire to add. We need to actually power the Octasonic board with 5V, so we do that by connecting one of the the Raspberry Pi 5V pins to the 5V pin on the Octasonic AVR header. This is the bottom left pin in the AVR header block (this is the 2 x 3 block on the top right of the board). See the attached photo showing where the AVR block is.

See the other attached diagram to find the 5V pin on the Raspberry Pi.

STEP 6: Install Software

Install Raspian

Start with a clean install of Raspbian Jessie, then update it to the latest version:

sudo apt-get update
sudo apt-get upgrade
Enable SPI

You must enable SPI on the Raspberry Pi for this project to work! Use the Raspberry Pi Configuration utility to do this.

It is also important to reboot the Pi after enabling SPI for it to take effect.

Install FluidSynth

Fluidsynth is an amazing free software MIDI synth. You can install it from the command-line with this command:

sudo apt-get install fluidsynth
Install the Rust Programming Language

The Ultrasonic Pi Piano is implemented in the Rust Programming Language from Mozilla (it's like C++ but without the bad bits). It's what all the cool kids are using these days.

Follow the instructions at https://rustup.rs/ to install Rust. To save you time, the instructions are to run this one command. You can accept the default answers to any questions during the install.

NOTE: Since posting this instructable, there are some issues with installing Rust on the Raspberry Pi. Bad timing :-/ but I have modified the command below to workaround the issue. Hopefully they will fix this soon. I am working on creating an image that people can download and burn onto an SD card. If you would like that, please contact me.

export RUSTUP_USE_HYPER=1<br>curl  https://sh.rustup.rs  -sSf | sh
Download the Ultrasonic Pi Piano source code

The source code for the Ultrasonic Pi Piano source code is hosted on github. There are two options for getting the code. If you are familiar with git and github, you can clone the repo:

git clone git@github.com:TheGizmoDojo/UltrasonicPiPiano.git

Alternatively, you can download a zip file of the latest code.

Compile the source code
cd UltrasonicPiPiano
cargo build --release
Test the code

Before we move onto making music in the next step, let's make sure that the software runs and that we can read valid data from the sensors.

Use the following command to run the application. This will read data from the sensors and translate them into MIDI notes that are then printed out on the console. As you move your hand over the sensors, you should see data being generated. If not, then skip to the troubleshooting section at the end of this instructable.

cargo run --release

If you are curious, the "--release" flag tells Rust to compile the code as efficiently as possible, as opposed to the default "--debug" setting.

STEP 7: Make Some Music!

Make sure you are still in the directory where you downloaded the source code and run the following command.

This "run.sh" script makes sure that the code has been compiled and then runs the code, piping the output into fluidsynth.

./run.sh

Make sure that you have amplified speakers connected to the 3.5mm audio jack on the Raspberry Pi and you should hear music as you move your hands over the sensors.

If you don't hear music and you have an HDMI monitor attached, then the audio output is probably going there instead. To fix this, simply run this command and then re-start the Pi Piano:

sudo amixer cset numid=3 1

Changing the volume

The volume (or "gain") is specified with the "-g" parameter to fluidsynth. You can modify the run.sh script and change this value. Please note that small changes in this parameter result in a big change in volume, so try increasing it by small amounts (like 0.1 or 0.2).

STEP 8: Gesture Control


See the video attached to this step for a full demonstration of the project, including how the gesture controls work.

The concept is very simple. The software keeps track of which sensors are covered (within 10cm) and which are not. This translates to 8 binary numbers (1 or 0). This is very convenient, since a sequence of 8 binary numbers makes a "byte" which can represent numbers between 0 and 255. If you don't already know about binary numbers then I highly recommend searching for a tutorial. Binary numbers are a fundamental skill to learn if you want to learn more about programming.

The software maps the current state of the sensors to a single byte representing the current gesture. If that number stays the same for a number of cycles, then the software acts on that gesture.

Because ultrasonic sensors aren't super reliable and there can be interference between sensors, you will need to exercise some patience when using the gestures. Try varying the distance that you hold your hands from the sensors as well as the angle that you hold your hands. You cold also try holding something flat and solid over the sensors to better reflect the sound.

STEP 9: Making an Enclosure


If you want to make this a permanent exhibit and be able to show it off to people, you'll probably want to make some kind of enclosure. This could be made from wood, cardboard, or many other materials. Here is a video showing the enclosure we are working on for this project. This is made from wood, with holes drilled out to hold the ultrasonic sensors in place.

STEP 10: Troubleshooting and Next Steps

Troubleshooting

If the project isn't working, it's usually down to a wiring error. Take your time to double check all the connections.

Another common issue is failing to enable SPI and rebooting the pi.

Please visit https://theotherandygrove.com/octasonic/ for full documentation including troubleshooting tips, with Rust and Python specific articles, and also information on how to get support.

Next Steps

Once you have the project working, I recommend experimenting with the code and trying out different musical instruments. MIDI instrument codes are between 1 and 127 and are documented here.

Do you want a single musical instrument with each sensor playing a different octave? Perhaps you'd like each sensor to be a separate instrument instead? The possibilities are almost limitless!

I hope you enjoyed this instructable. Please like it if you did, and be sure to subscribe to me here and to my YouTube channel to see future projects.

16 Comments

DanL209 5 years ago

Hello Andy,

I am working on building an art installment using a technique similar to the project you have explained so well above, excellent work! I was wondering if it is possible to set each specific sensor to play from different MIDI files. In addition, they would be MIDI files that I intend on creating myself so they would have to be integrated through FluidSynth in some way.

2019dfleck 5 years ago

Is there a way to adjust what instrument is played? I have looked online and there are a lot of different MIDI program numbers. My teacher and I are wondering if there is a way that we could set a specific instrument or sound to be played.

YogendranP 5 years ago

Hello Andy - am trying to recreate this Project with RPI3 and also i have bought the octasonic breakout board from Tindie store.

Can you please share with me the connection Diagram. I am confused with the detailed write vs your Video. In the Video 5V input is coming from RPi3 5V port and going to Logic level shifter HV Side.. however in the write it says 5V of RPI should be connected to the lower left end of AVR pins.

can you please share wiring diagram and pictures are not clear with Wiring.

Onlinea000 6 years ago

Hi,

I am trying to make this ultrasonic Pi Piano but when i connect just one ultrasonic sensor and all the other connection and run the code, it hangs at one of the

>noteoff 1 58

> noteon 1 59 127

> noteoff 1 59

and when i move my hand over the sensor nothing is detected. Plus the LED on the Octasonic board does not light up RED when I power it on. Please can you help

Thanks

Leena

AmineI5 6 years ago

hello , i have the same problem
do you have a solution .
thanks

rmoule 6 years ago

How would I use recorded voices instead of the music?

rmoule 6 years ago

I changed the instrument by going to the Wikipedia page and selecting number 23 - a harmonica. I changed this in the main.rs file in line - let mut instrum: Vec<u8> =vec! [1, 23, 18, 25, 41.....

But it didn't change. Do I need to change something else?

JaredS76 6 years ago

I see that you mentioned it in the instructions but I'm a bit confused on exactly where to change them in the code.

JaredS76 6 years ago

Any advice on how to change the sounds to different types of instruments? Your time is greatly appreciated!

lilysgrampa 6 years ago

SUCCESS!! I went back and checked all my wiring and found I'd flipped two wires.

lilysgrampa 6 years ago

Eve though git is installed the command

git@github.com:TheGizmoDojo/UltrasonicPiPiano.git

fails with: -bash: git@github.com:TheGizmoDojo/UltrasonicPiPiano.git: No such file or directory

I had to download and unzip it. You might want to check this or change the documentation.

lilysgrampa 6 years ago

Well i replaced the 5V pi to Octosonic and things are a little better, but still not running properly

lilysgrampa 6 years ago

So I have everything installed and when I run, it starts to play a scale and stops and sits there. Any ideas? here is the log I see:

pi@sonicpiano:~/UltrasonicPiPiano $ ./run.sh

Killing any already running pianos ...

Compiling code ...

Finished release [optimized] target(s) in 0.0 secs

Launching piano ...

FluidSynth version 1.1.6

Copyright (C) 2000-2012 Peter Hanappe and others.

Distributed under the LGPL license.

SoundFont(R) is a registered trademark of E-mu Systems, Inc.

fluidsynth: warning: Requested a period size of 64, got 256 instead

Type 'help' for help topics.

> # pin exported OK

> # set pin direction to IN

> # cm_per_note = 2

> # mode = Linear

> # instruments: [1, 10, 18, 25, 41, 89, 49, 14]

> prog 1 1

> prog 2 1

> prog 3 1

> prog 4 1

> prog 5 1

> prog 6 1

> prog 7 1

> prog 8 1

> noteon 1 48 127

> noteoff 1 48

> noteon 1 49 127

> noteoff 1 49

> noteon 1 50 127

> noteoff 1 50

> noteon 1 51 127

> noteoff 1 51

> noteon 1 52 127

> noteoff 1 52

> noteon 1 53 127

> noteoff 1 53

> noteon 1 54 127

> noteoff 1 54

> noteon 1 55 127

> noteoff 1 55

> noteon 1 56 127

> noteoff 1 56

> noteon 1 57 127

> noteoff 1 57

> noteon 1 58 127

> noteoff 1 58

> noteon 1 59 127

> noteoff 1 59

>

and I have to control-c it at this point. this is running on a Pi 3 using the 2017-07-05 full Jessie on a 16G SD card

goacego 7 years ago

nice job! one suggestion, you could use photo resistors instead of the ultrasonic sensors to reduce the cost greatly!