Sound Fruit - How to Make a Sculptural Audio Visualizer

Intro: Sound Fruit - How to Make a Sculptural Audio Visualizer

Sound Fruit is a project I did to study a free graphic scripting software called Processing. It consists of a little program that translates a mp3-file into a 3-dimensional object in .dxf format. The exported object is essentially an expression of sound, frozen in time. Depending on what sound you feed it, it will vary in its shape, giving every fruit a unique form. In the example shown I used the song "Rolling with the Punches" by The Blue Stones as a test.
The exported file can be opened in another 3d program, looked at, printed, or whatever you can think of! Potentially, you could print it hollow, wire it with an LED or two and make a cute little lamp.

Here is a video showing it in action:

Sound.Fruit - Sculptural Audio Visualizer from Tao C on Vimeo.

You need Processing to open the program. It's free, it's fun, and it can be downloaded here.
If you only want to use the program, you only need to follow the first 3 steps.

For people who are interested in the sketch itself, I'm explaining the details of the script starting from step 4. During my learning process I struggled quite a bit with finding good information about more advanced concepts and I had to piece together a lot of the information from obscure corners of the internet. I hope you will find this instructable useful in your own programming. This program is not a finished product yet and I'm still working on it, so if there are any suggestions or questions, feel free to comment. I'm barely more than a beginner myself, so any tips are greatly appreciated!
Many thanks to Brad Borevitz, who helped me a great deal in understanding programming and its application within the visual arts.

EDIT: I have added scale functionality and replaced the original .zip package with the updated one. Please redownload if you have the old version.

To scale down, press the 'n' key, to scale up, press 'm'.

STEP 1: Run the Sketch

Make sure you have Processing installed on your computer. Download the attached file, unzip it and you will see a number of .pde files. Double-click the one named "soundfruit_sketch.pde" to open the main file. In the sketch window, you will see a play-button in the top left corner. Click it to run the sketch.

STEP 2: How to Use the Program

The UI is very simple. it should look like it is pictured here, if nothing shows up, or if you just get a grey screen, close it and restart it. 

In the bottom right corner, there are two buttons. The play-button pauses/starts the music. The music will loop once it finishes. The record-button takes a "snapshot" of the object and exports it as a .dxf-file. Once it's recorded, the dxf-file will be in the main folder of the sketch. That's it. Simple as that. 

STEP 3: How to Imprint Your Own Sounds

In the main folder, you will find another folder named "data". This is the folder where all external data is stored. 
Go into the folder, and you should see a file named "punches.mp3". This is the file that is played when the program runs. The sound-data that you want to run has to be a .mp3-file, so convert it if it's not. You have 2 options to put it into the program:

1. Rename your file to "punches.mp3" and simply replace it with the one in the "data" folder.

2. Copy your file into the "data" folder, go into the sketch and change the following line:

          thinking = minim.loadFile("punches.mp3", 1024);  

          to

          thinking = minim.loadFile("yourfilename.mp3", 1024);

STEP 4: Setting Up Global Parameters

In the following steps I'm explaining the more complicated functions used in the sketch. Things such as how to set up a graphic button I have mostly omitted, as I am assuming that anyone interested in the next steps already knows basic Processing language. However, if there is interest in some of the lines that are not covered here, I am happy to explain these as well!



//The program uses 3 built-in libraries, the minim libraries are used for sound analysis, while the processing.dxf  library //allows you to "record" 3d objects. 

import ddf.minim.analysis.*;
import ddf.minim.*;
import processing.dxf.*;

//These parameters control how the fruit behaves and its resolution and appearance. 
int m = 6; //logavg 1 controls distribution of spectrum
int n = 6; //logavg 2 controls amplitude of spectrum
int ptDensity = 200; //density of circle around y-axis
float r; //radius

//Setup for the "scan" movement.
float beta = 0; //angle
int idx = 0; //index


//sound setup needed to play the mp3 file.
Minim       minim;
FFT         fft;
AudioInput in;
AudioPlayer thinking;

//The fruit is essentially a sphere that is divided into UV coordinates. This array stores the coordinates from each point on //the surface, allowing the fruit to memorize the highs and lows registered in the previous cycle.
PVector [][] coords;


The following parameters create the record button functionality.
RecordButton recIt;
int recX, recY, recSize;
boolean recHit = false;
boolean record = false;
color recFill, recHighlight;
int waveSize = 0;
boolean waveHit = false;
int waveFade = 255;

//play button setup
PlayButton playIt;
int playX, playY, playSize;
boolean playHit = false;
color playFill, playHighlight;
boolean play = true;

STEP 5: Setting Up the Program

void setup() {

//defines the size of program window and also that the program operates in a 3 dimensional space. 
  size(displayWidth, (displayHeight-20), P3D);


//sound creation - loads and plays the file included with the sketch. 
  minim = new Minim(this);
  thinking = minim.loadFile("punches.mp3", 1024);
  thinking.loop();
  fft = new FFT(thinking.bufferSize(), thinking.sampleRate());
  fft.logAverages(m, n);

  //In order to create a deformable sphere, I could not use the built-in sphere() function as it does not allow me to access //the point coordinates that make up the sphere. For that reason, I looked up the mathematical formula for spheres on //some math-discussion boards, which gave me the XYZ coordinates for every point, which would be stored in the array //that was set up before-hand.
  r = width/50; //radius
  beta = TWO_PI/ptDensity; //y-axis rotation
  coords = new PVector [ptDensity+1][fft.avgSize()];
  float x, y, z;
  for (int i = 0;i     float u = (i*TWO_PI/ptDensity);
    for (int j = 0;j       float v = (PI/coords[i].length)*j-PI/2;
      //      x = r * cos(theta);
      //      y = r * sin(theta);
      x=r* sin(u)* cos(v);
      y=r * cos(u) * cos(v);
      z= r * sin(v);

      coords[i][j] = new PVector(x, y, z);
      println(u + " " + v + " " + coords[i][j]);
    }
  }

  //playbutton parameters
  playX = width-100;
  playY = height-100;
  playSize = 30;
  ellipseMode(CENTER);
  playFill = color(5);
  playHighlight = color(200);
  //create play button
  playIt = new PlayButton();

  //recbutton parameters
  recX = width-60;
  recY = height-100;
  recSize = 30;
  //ellipseMode(CENTER);
  recFill = color(5);
  recHighlight = color(200);
  //create play button
  recIt = new RecordButton();
}

STEP 6: Shaping the Fruit

//advance index - This line shifts the current row on the sphere coordinate system by one row. only the current row registers the sound and imprints it onto the object. 
  idx = (idx+1)%(ptDensity);

  // calculate current row
  for (int i = 0; i<fft.avgSize(); i++) {

    float theta = (PI/fft.avgSize())*i-PI/2;


//the current row translates sound by picking up the sound spectrum and stretching it to fit on a half circle. the value at //each interval of the spectrum then translates into a magnitude change, radiating from the center of the sphere. 
    float add = (coords[idx][i].mag() + (0.05*(r+modFactor(i)*fft.getAvg(i))));

    coords[idx][i].setMag(add);
  }

//this double for-loop draws the half-circle and rotates it around the fruit, creating a scanner that adds to the distance from each point to the center.
  for (int j = 0; j<(ptDensity); j++) { //loop rotates half circle around y-axis to form sphere


//the beginShape() function spans all points on the object and creates the actual surface that makes up the object. 
    beginShape(QUAD_STRIP);
    for (int i = 0; i<fft.avgSize(); i++) { //this loop draws the half circle

      //      float theta = (PI/fft.avgSize())*i-PI/2;
      noStroke();
      //       stroke(map(fft.getAvg(i), 0, fft.avgSize(), 0, 256), 50, 55);
      fill(map(fft.getAvg(i), 0, fft.avgSize(), 0, 256), 50, 55);

      vertex(coords[j][i].x, coords[j][i].y, coords[j][i].z);

      if (j == ptDensity-1) {
        vertex(coords[0][i].x, coords[0][i].y, coords[0][i].z);
      }
      else {
        vertex(coords[j+1][i].x, coords[j+1][i].y, coords[j+1][i].z);
      } //create quadstrip-compatible geometry
    }
    endShape(CLOSE);
  }

Comments

[jrmasiero]

This is a great tutorial for those who want to make shapes from sound and then print it. Thank you.

[NataliaR7]

Hi! im super excited to use for meditation.. but i get this error .

Cannot find "ddf.minim.analysis" library. Line 3 in tab soundfruit_sketch

Cannot find "ddf.minim" library. Line 4 in tab soundfruit_sketch

Could you help me out?

im using Processing3.0b1 on a mac

[FidelisS]

Using Processing 3.0b6 here.

First thing was, that it gives an error ("cannot use size() here") so I changed that to fullScreen(P3D);

that worked and the Sketch loaded the UI with play and record button as well as your mp3-file and the visuals..

[Izola]

jreilly gavin, I used my own music, i.e. a recording done at a waterfall and it did not look so hot. It did not render a big image at all and it was as if it was dimmed. I think it is because of the intervals in the "music', it is not big enough like when you play a song with drums and rhythm, it is not an "even" sound like water. I then tried another song and got a similar look. I do not want a similar look, I want it different. I'd like to experiment with other colours and shapes. Any advice?

[Izola]

Is there a way to use other shapes, i.e. cubes or triangles? Also, I struggled to make a free-standing application after saving everything. Most people do not have processing installed. What alternatives are there?

[jreilly gavin]

Fantastic. I can't wait to play with it. Thanks!

Have you found that your favorite songs have a similar look? I'd be curious to see if an artist produces similar shaped 'fruit'.

So many possibilities thanks again. I totally see this taking up my spare time this week!

[mxfaiman]

Way Cool!

Is there any way to make it so you can scroll / pan in and out? As when it gets to a certain point it fills the screen and I'd like to be able to pan out.

[flesh_dog]

Certainly! The easiest method I can think of is to add a scale() function at the beginning of every void draw() loop. You would have to experiment with the scale-factor to find one that makes sense, I'm assuming it would be close to a factor of 1 (say 0.99 maybe). Basically, it will shrink the object a tiny little bit per loop and it will appear like as if it stays the same size throughout.

Alternatively, you could use one of the many libraries that enable camera control functionality in the processing window. (like this one: http://gdsstudios.com/processing/libraries/ocd/) I wasn't too concerned with this feature as the resulting .dxf object was more important to me.

[mxfaiman]

I have been playing with it but I'm completely new to coding and can't figure it out. I tried the scale but all it does is shrink the whole window and the ball still fills it up and no zoom. I also tried adding ocd to it and I got it to dolly but it added in a cube and focused on that which was nowhere near the ball. So how do I focus on the ball and delete the cube?

Thanks!

[flesh_dog]

hey mxfaiman,

I have added scale functionality as promised. Just redownload the zip package. Use the 'n' and 'm' keys to scale. Sorry for the delay!

[flesh_dog]

Hm. I can't say for sure without looking at your code. It sounds like as if you have inserted the function in the void setup(). Make sure it's inside the curly braces belonging to the void draw() function.

Alternatively, you could assign a key to the scale-function, so it only scales when you press the key on your keyboard. I will try to update the code with scaling functionality once I have some time. Sometime in the next few weeks.

[dneishtadt]

Hey, super cool! I was wondering, how do you output the result into a 3D file? DXF is 2D...

[flesh_dog]

Thanks! To my knowledge, DXF is often used for 2d applications like Illustrator/AutoCAD and also for 2d-based processes like laser-cutting, but it can store 3d information. Most 3d programs should support it. I used Rhino3d to import the dxf file, it automatically converts it into a mesh object. I'm pretty sure SketchUp supports dxf also, so that would be your free alternative.

[dneishtadt]

So you just open the dxf file in Rhino? hmm :)) never thought of that, will give it a try tomorrow

[JanaMarie]

Very nice!!

[craftclarity]

COOL!