I believe holographic musical instruments will be commonplace in the future, showing up everywhere from schools (for education), to homes (for fun), to media offices (for creativity), and in music studios (for production). The reason is simple: The holographic musical instrument takes a complex process and radically simplifies it: see another demo video here.
I'm using the term 'holographic music' to mean multidimensional musical structures mapped to 3d surfaces to be decoded through rotational motion. Just as optical holograms modulate light based on the 3D viewing angle, we can modulate sounds based on the relative 3D orientation of an object.
This is Part 1 of a 3-part series on a technology that I am calling the Dub Cadet: a holographic musical instrument. Part 1 will discuss theory and technical strategy, Part 2 will provide an arduino-based hardware solution, and Part 3 will explain the programming code that makes it work.
The hobby prototyping boards that I used in this project will rack up a bill of around $200. I invite you to visit my kickstarter page where I am offering DIY kits (with arduino source code) for $82. I have reduced the cost of this project significantly by building a custom circuit board that merges the microprocessor, gyroscope, midi decoder, and audio amplifier into one board :)
Step 1: The 3 Components of a Holographic Musical Instrument:
I am defining a holographic musical instrument as a device that tracks the angular rotation and angular momentum of an object to transform that data into live musical content.
Musical content subdivides into rhythmic variation (pulse) and harmonic variation (pitch) over time (speed). Thus we can isolate 3 groups of variables called pitch, pulse, and speed. In my solution, I am using compiled arduino code running on a micro-controller to do the data processing. An analog or hardware solution may also be possible.
I want to keep the definition loose in order to promote exploration. So in essence we have a rotational input feeding into some kind of processing system, and an output of musical notes that can change to produce musical progressions. My system breaks down into 3 technology solutions aimed at varying the pitch, pulse, and speed from a very minimal stream of angular data:
-Rotational Triad Synthesis: Builds musical chords of different pitches.
-Triple-Axis Rhythm Modulation: Creates interlocking rhythm templates.
-Instantaneous Time Formatting: Allows the alteration of musical speed in real-time.
In order to see the expansive possibilities of generative holographic music, we need to take a look at the theory and application of these three solutions. Please understand, that some background knowledge is required to delve into these concepts, but no experience is actually needed to effectively operate the instrument. Like a guided missile, all the technology needed to hit a target runs silently inside.
Ideally, but optionally, a holographic musical instrument should also provide visual feedback (to offer a fully immersive experience). I have designed a stunning circular layout of RGB LED's that display the musical data as visual pulses of colored light. I will be talking more about this in the next instructable installment (Part 2).
I will re-iterate that this is a practical look into a working solution, a jumping off point into a whole world of exploration and development. Actual code examples and hardware solutions will be offered in parts 2 and 3!
Step 2: Rotational Triad Synthesis:
The triad is the most basic harmonic chord while also providing all the information needed to create interesting music. It consists of three notes that work together to produce a more complex musical sound.
By looking at the compound rotation of a ball in terms of it's x, y, and z components, we naturally arrive at 3 coordinates that are each used to pick notes from the scale through linear (1:1) mapping. This means that the same rotational value creates the same note every time.
We can also think of the rotational instrument as three superimposed pitch wheels that can be adjusted independently or all at the same time, depending on how you move the sphere. In any case, it is the format of the ball that allows so many new possibilities to emerge. Even if you have 3 pitch wheels on a synthesizer, you only have two hands so operating three simultaneous controls would not be ideal.
By providing special rules as to what notes can be used and which ones to avoid (in the form a scale selection) we can create the conditions for harmonious musical emergence. Through experimentation, I have found that a scale of 22 to 24 notes provides just the right variety for pleasant and diverse musical creations.
At any one time, we can form one of 12,144 triads (24 * 23 * 22) contained in our musical scale of 24 notes. But a problem of aesthetics does arise. Some combinations do not sound harmonious. To solve this problem, I have created a dissonance rejection algorithm that detects extreme dissonances (harsh or competing adjacent notes) and rounds the offending selection to a more harmonious triad before being processed into sound. With MIDI notes, this is fairly easy to do. By analyzing a given chord we can find instances where the notes are only 1 MIDI number apart and simply shift the upper and lower part of the triad to make them at least 2 steps apart.
Dissonance rejection reduces the number of available triads to about 11,000 (depending on the scale being used) providing a pool of every aesthetically useful triad contained in our scale. This is what makes it exciting: every possible combination is available to you in a real time experience of tactile navigation.
The natural motion of a ball lends itself to gradual orientation changes that create a progression of notes with harmonic and melodic flow and consistency. Since most rotations involve turning the ball over two dominant axi, a relationship of continuity is formed where we can hear the notes shifting into new chords, in a gradual way. This gradual harmonic shifting also happens in most of the popular and classical music we enjoy. The physical shape of the sphere thus enables the most freedom of movement and musical content while also providing natural limits to the rate of continuous and simultaneous change.
The rotation of a sphere is also cyclic, meaning that as you pass 360 degrees, you roll back into the initial position again. Cyclic motion is also very common in music so the correlation of physical cyclic motion translates well into musical progressions. For example, many great musicians play free-form solos while continually referencing the root note of the scale. They remind us of where the musical "home" is and take us on a journey through the scale. With a spherical controller, it is very easy to do this intuitively.
Step 3: Triple-Axis Rhythm Modulation:
First, it is important to understand that the orientation of each axis is being mapped through a coded system to select rhythm options in the same way that genes code the diversity of biological organisms.
By changing the rotational orientation of the sphere, we "dial in" a new rhythm combination for that particular moment. This get's interesting when the sphere is moving because it means you are dynamically shifting through trillions of rhythm possibilities by simply turning the ball.
Rhythm is a sequence of pulses and the spaces (rests) between those pulses. If we consider the possible combinations of pulses and rests in a sequence of four units, there are 16 combinations we can generate (2^4). These 16 combinations are the musical building blocks, or DNA used to build all of the sounds in the Dub Cadet experience. A bar of music consists of 8 of these 4-unit building blocks in sequence.
The Dub Cadet breaks one measure or bar of music into 32 pulses per bar (8 units of musical DNA) for a startling 4,294,967,296 rhythmic possibilities (2^32). Since we are selecting a separate rhythm for each axis, we must take the third power of 4,294,967,296 to find out just how many options are possible in this system. I am only going through this calculation because the actual number of possibilities is astounding. The answer is: 79,228,162,514,264,337,593,543,950,336, which is cool, but annoying humungous. It is also impressive to know that the compiled code required to generate this kind of variation weighs in at about 12 kilobytes or about the size of a scam email. We also run into an issue of aesthetics; not all of those options are aesthetically pleasing!
Through trail and error, I have reduced the number of possibilities for the lowest note from 16 to 5. The bass note of a triad is traditionally the rhythmic backbone and by reducing its range of positions, we can maintain a sense of balance in the rhythm. Likewise, the middle note of our triad has been reduced to 10 possible rhythms. I've left the highest note of our triad to exist in any of the 16 positions, since it is traditional for the higher voices and instruments to be given the most expressive freedom.
This leads us to a new calculation of (2^5 * 2^10 * 2 ^15) = (32 * 1024 * 4,294,967,296) = 140,737,488,355,328 possible rhythms for each and every bar of music.
This generative system is great for breaking out of the over-used box of musical tricks to discover new ways to pulse the sound. If you listen to the radio for about an hour you will likely hear the same 20 or 30 rhythms being played at different speeds. The exposure to new possible sounds quickly breaks the mind from the common patterns, offering creative stimulation and uncapped enrichment.
Step 4: Instantaneous Time Formatting
Instantaneous Time Formatting is the ability to direct the tempo and pace of the music by spinning the ball at different rates. It is common knowledge that a musical conductor waves a wand to direct the pace of the music. The motion sensors of the Dub Cadet allow us to detect the precise the motion of the ball on each axis and create interlocking rhythms that wax and wane with the motions of the user.
The gyroscopic sensors being used measure the angular rate (speed of the spin) of each axis. This means we get a constant readout of the actual speed that the ball is rotating about it's x, y, and z axi. The Dub Cadet analyzes these angular speeds and triggers notes in correlation to them.
By using quantization, or snapping the notes that are triggered to a common grid, we maintain the ability to dynamically ajust the speed of note deployment, while maintaining an overall rhythmic structure. In other words, we keep the tempo locked to a common grid, while allowing the momentum of the sphere to position notes on that grid.
When the speed is zero, we maintain a period of rest. As the device begins to read higher rotational values, it triggers sounds at a faster and faster rate. Keep in mind that the quantized note positions are like slots in an ice-cube tray that are only filled with water when the speed and orientation requires. This means we can always trigger the note to be on-time and on beat, no matter how fast or slow the ball is moving. Thus we arrive at a solution of additive interlocking design. By moving faster we generate more notes that play fast and by moving slow we generate less notes that last longer. My current solution isolates the length of the note from the time it is triggered for a truly expressive experience.
In summery, I have provided the theory of the technologies I have created for turning rotational movement into music. Stay tuned for the next installment which will cover building a system of electronics to put these concepts into practical application.
And if you haven't done so yet, check out my kickstarter page!