Sound Fruit - How to Make a Sculptural Audio Visualizer

Sound Fruit is a project I did to study a free graphic scripting software called Processing. It consists of a little program that translates a mp3-file into a 3-dimensional object in .dxf format. The exported object is essentially an expression of sound, frozen in time. Depending on what sound you feed it, it will vary in its shape, giving every fruit a unique form. In the example shown I used the song "Rolling with the Punches" by The Blue Stones as a test.
The exported file can be opened in another 3d program, looked at, printed, or whatever you can think of! Potentially, you could print it hollow, wire it with an LED or two and make a cute little lamp.

Here is a video showing it in action:

Sound.Fruit - Sculptural Audio Visualizer from Tao C on Vimeo.

You need Processing to open the program. It's free, it's fun, and it can be downloaded here.
If you only want to use the program, you only need to follow the first 3 steps.

For people who are interested in the sketch itself, I'm explaining the details of the script starting from step 4. During my learning process I struggled quite a bit with finding good information about more advanced concepts and I had to piece together a lot of the information from obscure corners of the internet. I hope you will find this instructable useful in your own programming. This program is not a finished product yet and I'm still working on it, so if there are any suggestions or questions, feel free to comment. I'm barely more than a beginner myself, so any tips are greatly appreciated!
Many thanks to Brad Borevitz, who helped me a great deal in understanding programming and its application within the visual arts.

EDIT: I have added scale functionality and replaced the original .zip package with the updated one. Please redownload if you have the old version.

To scale down, press the 'n' key, to scale up, press 'm'.

Step 1: Run the Sketch

Make sure you have Processing installed on your computer. Download the attached file, unzip it and you will see a number of .pde files. Double-click the one named "soundfruit_sketch.pde" to open the main file. In the sketch window, you will see a play-button in the top left corner. Click it to run the sketch.

Step 2: How to Use the Program

The UI is very simple. it should look like it is pictured here, if nothing shows up, or if you just get a grey screen, close it and restart it. 

In the bottom right corner, there are two buttons. The play-button pauses/starts the music. The music will loop once it finishes. The record-button takes a "snapshot" of the object and exports it as a .dxf-file. Once it's recorded, the dxf-file will be in the main folder of the sketch. That's it. Simple as that. 

Step 3: How to Imprint Your Own Sounds

In the main folder, you will find another folder named "data". This is the folder where all external data is stored. 
Go into the folder, and you should see a file named "punches.mp3". This is the file that is played when the program runs. The sound-data that you want to run has to be a .mp3-file, so convert it if it's not. You have 2 options to put it into the program:

1. Rename your file to "punches.mp3" and simply replace it with the one in the "data" folder.

2. Copy your file into the "data" folder, go into the sketch and change the following line:

          thinking = minim.loadFile("punches.mp3", 1024);  

          to

          thinking = minim.loadFile("yourfilename.mp3", 1024);

Step 4: Setting Up Global Parameters

In the following steps I'm explaining the more complicated functions used in the sketch. Things such as how to set up a graphic button I have mostly omitted, as I am assuming that anyone interested in the next steps already knows basic Processing language. However, if there is interest in some of the lines that are not covered here, I am happy to explain these as well!



//The program uses 3 built-in libraries, the minim libraries are used for sound analysis, while the processing.dxf  library //allows you to "record" 3d objects. 

import ddf.minim.analysis.*;
import ddf.minim.*;
import processing.dxf.*;

//These parameters control how the fruit behaves and its resolution and appearance. 
int m = 6; //logavg 1 controls distribution of spectrum
int n = 6; //logavg 2 controls amplitude of spectrum
int ptDensity = 200; //density of circle around y-axis
float r; //radius

//Setup for the "scan" movement.
float beta = 0; //angle
int idx = 0; //index


//sound setup needed to play the mp3 file.
Minim       minim;
FFT         fft;
AudioInput in;
AudioPlayer thinking;

//The fruit is essentially a sphere that is divided into UV coordinates. This array stores the coordinates from each point on //the surface, allowing the fruit to memorize the highs and lows registered in the previous cycle.
PVector [][] coords;


The following parameters create the record button functionality.
RecordButton recIt;
int recX, recY, recSize;
boolean recHit = false;
boolean record = false;
color recFill, recHighlight;
int waveSize = 0;
boolean waveHit = false;
int waveFade = 255;

//play button setup
PlayButton playIt;
int playX, playY, playSize;
boolean playHit = false;
color playFill, playHighlight;
boolean play = true;

Step 5: Setting Up the Program

void setup() {

//defines the size of program window and also that the program operates in a 3 dimensional space. 
  size(displayWidth, (displayHeight-20), P3D);


//sound creation - loads and plays the file included with the sketch. 
  minim = new Minim(this);
  thinking = minim.loadFile("punches.mp3", 1024);
  thinking.loop();
  fft = new FFT(thinking.bufferSize(), thinking.sampleRate());
  fft.logAverages(m, n);

  //In order to create a deformable sphere, I could not use the built-in sphere() function as it does not allow me to access //the point coordinates that make up the sphere. For that reason, I looked up the mathematical formula for spheres on //some math-discussion boards, which gave me the XYZ coordinates for every point, which would be stored in the array //that was set up before-hand.
  r = width/50; //radius
  beta = TWO_PI/ptDensity; //y-axis rotation
  coords = new PVector [ptDensity+1][fft.avgSize()];
  float x, y, z;
  for (int i = 0;i     float u = (i*TWO_PI/ptDensity);
    for (int j = 0;j       float v = (PI/coords[i].length)*j-PI/2;
      //      x = r * cos(theta);
      //      y = r * sin(theta);
      x=r* sin(u)* cos(v);
      y=r * cos(u) * cos(v);
      z= r * sin(v);

      coords[i][j] = new PVector(x, y, z);
      println(u + " " + v + " " + coords[i][j]);
    }
  }

  //playbutton parameters
  playX = width-100;
  playY = height-100;
  playSize = 30;
  ellipseMode(CENTER);
  playFill = color(5);
  playHighlight = color(200);
  //create play button
  playIt = new PlayButton();

  //recbutton parameters
  recX = width-60;
  recY = height-100;
  recSize = 30;
  //ellipseMode(CENTER);
  recFill = color(5);
  recHighlight = color(200);
  //create play button
  recIt = new RecordButton();
}

Step 6: Shaping the Fruit

//advance index - This line shifts the current row on the sphere coordinate system by one row. only the current row registers the sound and imprints it onto the object. 
  idx = (idx+1)%(ptDensity);

  // calculate current row
  for (int i = 0; i<fft.avgSize(); i++) {

    float theta = (PI/fft.avgSize())*i-PI/2;


//the current row translates sound by picking up the sound spectrum and stretching it to fit on a half circle. the value at //each interval of the spectrum then translates into a magnitude change, radiating from the center of the sphere. 
    float add = (coords[idx][i].mag() + (0.05*(r+modFactor(i)*fft.getAvg(i))));

    coords[idx][i].setMag(add);
  }

//this double for-loop draws the half-circle and rotates it around the fruit, creating a scanner that adds to the distance from each point to the center.
  for (int j = 0; j<(ptDensity); j++) { //loop rotates half circle around y-axis to form sphere


//the beginShape() function spans all points on the object and creates the actual surface that makes up the object. 
    beginShape(QUAD_STRIP);
    for (int i = 0; i<fft.avgSize(); i++) { //this loop draws the half circle

      //      float theta = (PI/fft.avgSize())*i-PI/2;
      noStroke();
      //       stroke(map(fft.getAvg(i), 0, fft.avgSize(), 0, 256), 50, 55);
      fill(map(fft.getAvg(i), 0, fft.avgSize(), 0, 256), 50, 55);

      vertex(coords[j][i].x, coords[j][i].y, coords[j][i].z);

      if (j == ptDensity-1) {
        vertex(coords[0][i].x, coords[0][i].y, coords[0][i].z);
      }
      else {
        vertex(coords[j+1][i].x, coords[j+1][i].y, coords[j+1][i].z);
      } //create quadstrip-compatible geometry
    }
    endShape(CLOSE);
  }