Ultrasonic Pi Piano With Gesture Controls!

This project uses inexpensive HC-SR04 ultrasonic sensors as inputs and generates MIDI notes that can be played through a synthesizer on the Raspberry Pi for a high quality sound.

The project also uses a basic form of gesture control, where the musical instrument can be changed by holding your hands over the two outermost sensors for a few seconds. Another gesture can be used to shut down the Raspberry Pi once you are finished.

The video above shows the finished product in a 3D-printed enclosure. There is a more in-depth video later in this instructable that explains how the project works.

Please also check out the latest documentation and tutorials here, which now includes information on a Python version of this project (this instructable uses the Rust programming language).

You can also buy the Ultrasonic Pi Piano as a kit here.

Step 1: Ingredients

For this instructable, you will need the following ingredients:

Raspberry Pi (2 or later) with SD card
8 HC-SR04 ultrasonic sensors
Octasonic Breakout Board
Bi-Directional Logic Level Converter
8 x 4-pin molex jumper wires (female-female) for connecting the ultrasonic sensors to the Octasonic breakout
13 x 6" Female-Female Jumper Wires for connecting the Raspberry Pi, Octasonic, and Logic Level Converter
Suitable power supply for the Raspberry Pi
PC speakers or similar

I would recommend using a Raspberry Pi 3 if possible since it has more computing power, resulting in a more responsive and pleasing sound. It can work fine with a Raspberry Pi 2 with a little tweaking but I would not attempt to use the original Raspberry Pi for this project.

HC-SR04 ultrasonic sensors have 4 connections - 5V, GND, Trigger, and Echo. Typically, Trigger and Echo are connected to separate pins on a microcontroller or Raspberry Pi but that means you would need to utilize 16 pins to connect 8 sensors, and this is not practical. This is where the Octasonic breakout board comes in. This board connects to all of the sensors and has a dedicated microcontroller that monitors the sensors and then communicates with the Raspberry Pi over SPI.

HC-SR04 require 5V and the Raspberry Pi is only 3.3V, so this is why we also need the logic level converter that will connect the Raspberry Pi to the Octasonic breakout board.

Step 2: Connect the Ultrasonic Sensors to the Octasonic Board

Use 4 female-female jumper wires to connect each ultrasonic sensor to the board, being careful to connect them the correct way around. The board is designed so that the pins are in the same order as the pins on the ultrasonic sensor. From left to right on the board, the pins are GND, Trigger, Echo, 5V.

Step 3: Connect the Logic Level Converter to the Octasonic Board

The Raspberry Pi and Octasonic Board communicate over SPI. SPI uses 4 wires:

Master In, Slave Out (MISO)
Master Out, Slave In (MOSI)
Serial Clock (SCK)
Slave Select (SS)

Additionally, we need to connect power (5V and GND).

The logic level converter has two sides - a low voltage (LV) and a high voltage (HV). The Raspberry will connect to the LV side since it is 3.3V. The Octasonic will connect to the HV side since it is 5V.

This step is for connecting the Octasonic to the HV side of the logic level converter.

See the photo attached to this step showing which pins should be connected to the logic level converter.

The connections from the Octasonic to the Logic Level converter should be as follows:

5V to HV
SCK to HV4
MISO to HV3
MOSI to HV2
SS to HV1
GND to GND

Step 4: Connect the Logic Level Converter to the Raspberry Pi

The Raspberry Pi and Octasonic Board communicate over SPI. SPI uses 4 wires:

Master In, Slave Out (MISO)
Master Out, Slave In (MOSI)
Serial Clock (SCK)
Slave Select (SS)

Additionally, we need to connect power (3.3V and GND). The logic level converter has two sides - a low voltage (LV) and a high voltage (HV). The Raspberry will connect to the LV side since it is 3.3V. The Octasonic will connect to the HV side since it is 5V.

This step is for connecting the Raspberry Pi to the LV side of the logic level converter.

The connections from the Raspbery Pi to the Logic Level converter should be as follows:

3.3V to LV
GPIO11 (SPI_SCLK) to LV4
GPIO09 (SPI_MISO) to LV3
GPIO10 (SPI_MOSI) to LV2
GPIO08 (SPI_CE0_N) SS to LV1
GND to GND

Use the diagram attached to this step to locate the correct pins on the Raspberry Pi!

Step 5: Connect Raspberry Pi 5V to Octasonic 5V

There is one final wire to add. We need to actually power the Octasonic board with 5V, so we do that by connecting one of the the Raspberry Pi 5V pins to the 5V pin on the Octasonic AVR header. This is the bottom left pin in the AVR header block (this is the 2 x 3 block on the top right of the board). See the attached photo showing where the AVR block is.

See the other attached diagram to find the 5V pin on the Raspberry Pi.

Step 6: Install Software

Install Raspian

Start with a clean install of Raspbian Jessie, then update it to the latest version:

sudo apt-get update
sudo apt-get upgrade

Enable SPI

You must enable SPI on the Raspberry Pi for this project to work! Use the Raspberry Pi Configuration utility to do this.
It is also important to reboot the Pi after enabling SPI for it to take effect.

Install FluidSynth

Fluidsynth is an amazing free software MIDI synth. You can install it from the command-line with this command:

sudo apt-get install fluidsynth

Install the Rust Programming Language

The Ultrasonic Pi Piano is implemented in the Rust Programming Language from Mozilla (it's like C++ but without the bad bits). It's what all the cool kids are using these days.

Follow the instructions at https://rustup.rs/ to install Rust. To save you time, the instructions are to run this one command. You can accept the default answers to any questions during the install.

NOTE: Since posting this instructable, there are some issues with installing Rust on the Raspberry Pi. Bad timing :-/ but I have modified the command below to workaround the issue. Hopefully they will fix this soon. I am working on creating an image that people can download and burn onto an SD card. If you would like that, please contact me.

export RUSTUP_USE_HYPER=1<br>curl  https://sh.rustup.rs  -sSf | sh

Download the Ultrasonic Pi Piano source code

The source code for the Ultrasonic Pi Piano source code is hosted on github. There are two options for getting the code. If you are familiar with git and github, you can clone the repo:

git clone git@github.com:TheGizmoDojo/UltrasonicPiPiano.git

Alternatively, you can download a zip file of the latest code.

Compile the source code

cd UltrasonicPiPiano
cargo build --release

Test the code

Before we move onto making music in the next step, let's make sure that the software runs and that we can read valid data from the sensors.

Use the following command to run the application. This will read data from the sensors and translate them into MIDI notes that are then printed out on the console. As you move your hand over the sensors, you should see data being generated. If not, then skip to the troubleshooting section at the end of this instructable.

cargo run --release

If you are curious, the "--release" flag tells Rust to compile the code as efficiently as possible, as opposed to the default "--debug" setting.

Step 7: Make Some Music!

Make sure you are still in the directory where you downloaded the source code and run the following command.

This "run.sh" script makes sure that the code has been compiled and then runs the code, piping the output into fluidsynth.

./run.sh

Make sure that you have amplified speakers connected to the 3.5mm audio jack on the Raspberry Pi and you should hear music as you move your hands over the sensors.

If you don't hear music and you have an HDMI monitor attached, then the audio output is probably going there instead. To fix this, simply run this command and then re-start the Pi Piano:

sudo amixer cset numid=3 1

Changing the volume

The volume (or "gain") is specified with the "-g" parameter to fluidsynth. You can modify the run.sh script and change this value. Please note that small changes in this parameter result in a big change in volume, so try increasing it by small amounts (like 0.1 or 0.2).

Step 8: Gesture Control

See the video attached to this step for a full demonstration of the project, including how the gesture controls work.

The concept is very simple. The software keeps track of which sensors are covered (within 10cm) and which are not. This translates to 8 binary numbers (1 or 0). This is very convenient, since a sequence of 8 binary numbers makes a "byte" which can represent numbers between 0 and 255. If you don't already know about binary numbers then I highly recommend searching for a tutorial. Binary numbers are a fundamental skill to learn if you want to learn more about programming.

The software maps the current state of the sensors to a single byte representing the current gesture. If that number stays the same for a number of cycles, then the software acts on that gesture.

Because ultrasonic sensors aren't super reliable and there can be interference between sensors, you will need to exercise some patience when using the gestures. Try varying the distance that you hold your hands from the sensors as well as the angle that you hold your hands. You cold also try holding something flat and solid over the sensors to better reflect the sound.

Step 9: Making an Enclosure

If you want to make this a permanent exhibit and be able to show it off to people, you'll probably want to make some kind of enclosure. This could be made from wood, cardboard, or many other materials. Here is a video showing the enclosure we are working on for this project. This is made from wood, with holes drilled out to hold the ultrasonic sensors in place.

Step 10: Troubleshooting and Next Steps

Troubleshooting

If the project isn't working, it's usually down to a wiring error. Take your time to double check all the connections.

Another common issue is failing to enable SPI and rebooting the pi.

Please visit https://theotherandygrove.com/octasonic/ for full documentation including troubleshooting tips, with Rust and Python specific articles, and also information on how to get support.

Next Steps

Once you have the project working, I recommend experimenting with the code and trying out different musical instruments. MIDI instrument codes are between 1 and 127 and are documented here.

Do you want a single musical instrument with each sensor playing a different octave? Perhaps you'd like each sensor to be a separate instrument instead? The possibilities are almost limitless!

I hope you enjoyed this instructable. Please like it if you did, and be sure to subscribe to me here and to my YouTube channel to see future projects.