Introduction: Melody

By

Alongside the many advantages and technological solutions that allow work from home, the difficulty of formulating and creating life support among co-workers remains. MELODY is a digital-physical device that enables the creation of collaborative short musical jams. Co-workers coordinate time and the device sets a jam session with turns and different random sounds. The first participant sets a specific rhythm, after which each participant adds their own musical section corresponding to the set rhythm. In order to make it easier for users with no musical background, the software helps them keep pace by sampling their clicks and adjusting to the appropriate rhythm. The session ends after about 3 minutes when all participants have finished recording their part.

How does it work?

Melody is based on ESP2866 hardware, which communicates with a Node-Red server over MQTT protocol. The device translates the player’s notes into a string of characters that is sent to the server and from the server back to the other players. This allows everyone to play and hear the tune without interruption from their network connection.

Melody has two main visual indicators. The first is a LED strip that lets the player know when Loop starts and when it ends and indicates if it’s the player’s turn. The second is a LED display in the center of the product, which is used to visually display the existing tune. A countdown from 3 to 1 indicates to start playing and a timings display directs the user when and how she wants to contribute to the group’s Melody. The recording is automatically saved to the company’s cloud for future use.

This project was designed by four students at the Media innovation lab (MiLab) at the interdisciplinary Center Herzliya (IDC): Shahar Agassy, Eden Bar-Tov, Gal Eshchar, and Gad Stern. With the assistance of Zvika Markfeld, Netta Ofer and Michal Leschinsky and the guidance of Noa Morag and Oren Zuckerman.

thanks to Tom Granot for creating a great instructable that helped me learn how to implement some of the things here (some of the steps here are modeled after this great instructable).

Supplies

Step 1: Understanding the Flow

In this project we tried to tackle some issues:

  1. How can we make it online, so players can play at the same time?
  2. How can we go around internet latencies and create a seamless experience?
  3. How can we make the music sound good even for people without musical background?

Timing and serialization of the music

In order to solve the first issue we looked at the MIDI protocol and tried to use it, but we saw that it is more robust then what we actually needed also we really wanted to make it simple so we can build the first working prototype. So we took inspiration from the MIDI and made our musical loop represented by a string of numbers (from 0-5) times the loops size time the players (we will explain all the musical math later).

In music, we split the rhythms into musical bars. Each bar is basically a small-time segment
we choose to use 4/4 (meaning 4 beats in a musical bar) - the most common one.

Each beat is then split into 4 sampling windows so every note played will be auto aligned to a c good position and also allow us to represent a song as a string of numbers to send to the server.

In order to be friendly to players without a musical background we did three things:

  1. Limit the number of keys to make the player focus on fewer options.
  2. We picked notes on the same scale that play well together so there won't be any dissonance sound.
  3. Each press is set to the "window" of the rhythm thus alien the player music to the rhythm

Communication protocols

So after we understood the logic behind the music, how can we communicate it between our players?

for that, we use MQTT, a publish-subscribe network protocol that transports messages between devices.

each player is subscribed to two topics: the loop (get the most current loop) and the turn (gets the id of the current player for sync purposes).

In their turn when a player is done playing the tune he/she will press the UP button and the loop (the updated one) will be sent to the MQTT broker, which will transmit it back to all the players on the loop channel.

this loop will stay "dormant" until the current loop is done playing and then will replace it. thus it will be transparent to the player. also since the new loop is currently saved locally at the player device there is no internet latency for the music an so we solved the second issue.

Step 2: Setting the Server - Ngrok

ngrok is a tunneling service. It allows us to expose a locally running service (in our case, Node-RED) to the outside world - without the hassle of setting up a server or dealing with DNS records. You simply run Node-RED on your computer, and then run ngrok on the same port Node-RED is running on.

That's it - you'll get a URL that you can use to access Node-RED from anywhere in the world, regardless of what network it's connected to.

Installation & Configuration

  1. Download ngrok for your operating system from here.
  2. Follow the step on the download page, up until the "Fire it up" step.
  3. In the "Fire it up step", swap the 80 for 1883 - and the http to tcp as in, ./ngrok tcp 1883 depending on your
  4. save the URL and port number (has seen in the image) we will need the, later.

Step 3: Setting the Server - Node-Red

The Server logic of the project, Node-RED is a visual programming environment that allows you to connect various software (and hardware!).

Here we made the logic of the communication between all the players (sharing and receiving the loops and coordinating the turns)

Install of Node-Red

follow the following steps to load our Node-RED flow on your local computer:

  1. Node-RED requires Node.js, install it from here
  2. install Node-RED itself using the instructions here.

Now that you have Node-RED installed, run it using the instructions on the step above and validate you can see an empty canvas page. It should be located in http://127.0.0.1:1880

You'll now need to import the flow that we used for this project, you can find it here and just press import add the JSON file and press Deploy.

Install of Node-Red:

if you look at the image that is attached to this step you can see we have 2 main "actions" we receive a current loop from one of our players and then we transmit it to all the other players. in addition, we broadcast the new turn to all the players. so the game stays in sync.

Step 4: Setting the Server - MQTT (Mosquitto)

Since Node-RED does not have its own MQTT broker, and we will need to communicate with our sensors and activators over MQTT, we'll use a dedicated MQTT broker. Since Node-RED recommends Mosquitto, this is the one we will use. See here for some information about MQTT and why it's often used in IoT projects.

Installation & Configuration

  1. Download Mosquitto from here and install it, all according to your operating system.
  2. Normally, you'd need to follow the instructions here to connect Node-RED to Mosquitto. However, if you used our flow, it's already pre-configured for you. As long as you install the flow and Mosquitrro properly, and Mosquitto runs on port 1883 (on which it runs by default), it should work out of the box.
  3. Note that this means that the MQTT broker and your Node-RED server run on the same machine. This is useful for simplifying communication inside the system. See the note below for more information.

Monitoring MQTT traffic

I used MQTTfx to monitor the traffic, it's a great tool with a very simple GUI.

Step 5: The Code

you can find the code in GitHub (with all the data files and the config.h)

Dependencies:

before loading the code to the esp2866 you'll need to install a few libraries:

  1. libmad-8266 (decode the music from SPIFF and into I2S)
  2. EspMQTTClient
  3. ESP8266WiFi
  4. Adafruit_NeoPixel

Upload the sounds to the ESP using SPIFF:

  1. follow this great instructable.
  2. add the data folder to the source code directory.
  3. In the Arduino IDE under Tools change Flash size to "4MB (FS:3MB TOA:~512KB)"
  4. Also under Tools Press ESP2866 Sketch Data Upload


Setting the parameters:

after that go to the config.h file and add the required data such as WIFI credentials and the ngrok URL and port from the previous step (check the attached photo for reference).

p.s - I've yet added an auto-connect feature to help you set the WIFI and ngrok data from your smart-phone, since it was just a first proof of concept, I would like to add it some-day.

Set the amount of player you wish (this game works best for 2-3 players and out of the box it is loaded with an array of sounds for 2 players). but can be easily adjusted for more:

for each player add another flow in the node-red to publish a loop in an under a user-specific topic.

also, you can edit the musical sound by canging this array to your custom sounds:

here you can see 3 kinds of instruments (Chrods for player 0, Lead for player 1, and Bass for player 2)

const char* paths[NUMofNotes] = {"/blank1.wav", "/Chords_Am.wav", "/Chords_F.wav", "/Chords_C.wav", "/Chords_G.wav", "/Chords_Dm.wav", "/blank2.wav", "/Lead_C.wav", "/Lead_D.wav", "/Lead_E.wav", "/Lead_G.wav", "/Lead_A.wav", "/blank0.wav", "/Bass_C3.wav", "/Bass_D3.wav", "/Bass_F3.wav", "/Bass_G3.wav", "/Bass_A3.wav"};

Step 6: Print the 3D Model

For the first step, download the STL and print them.

after removing the supports and maybe a little bit of sanding (depending on the resolution of the printer)

paint it to the desired color

Attachments

Step 7: Assembling and Welding

So basically here is where the real magic happens.

you can follow these schematics and weld everything together.

mind that you can change the position of the PINs just remember to change it in the code as well.

the A0 and the I2S are pretty fixed in place:

since the A0 is for the resistor bridge ( we use the difference in the current to know which button out of the 5 was pressed - similar to this Instructables.

the I2S has a specific coding you can find it here

Step 8: Play Some Loops With Your Friends