Data Logging Sensors/Inputs With Processing

Intro: Data Logging Sensors/Inputs With Processing

Simple guide to using processing software to log data from your Arduino through the Serial port. The data will be log and save as a csv table for easy of use with other software.

Processing is used for a variety of projects, from visual arts to research.

Here is the website for Processing: https://processing.org/

They have simple tutorials to get you started and a reference page to help with understanding the code.

Items you will need:

Arduino Software(including attached sensor of your choice) - We will be using the button project from my previous instructable to aid in this project, but a simple button input, like the StateChange example Arduino provides, will also work.

Processing Software

~We will skip over the Arduino hook-up part, and go straight to the the code and output.

STEP 1: Setup the Arduino With Code

This is a simple code that output the number one to the serial port as a string.

The output will be what we collect with processing. Ideally you want to have it output as a string otherwise Processing will not record the number or letter that is output, but the DEC number of the value.

You can find out more about DEC number at: http://www.asciitable.com/

~You can program either Arduino to output string values or program Processing to convert values to string.

For this example we will have Arduino output as a string.

STEP 2: Processing Code for Data Collection

This first bit of code will make Processing collect data and store it as a csv file .

Before starting Processing, make sure the Arduino sketch is uploaded to your Arduino, but do not open the serial monitor. If the serial monitor is open Processing will not be able to access the COM port and collect data.

//import the required libraries
import processing.serial.*;
Serial mySerial;
Table table;

void setup()
{
  //set mySerial to listen on COM port 10 at 9600 baud
  mySerial = new Serial(this, "COM10", 9600);
  
  table = new Table();
  //add a column header "Data" for the collected data
  table.addColumn("Data");
}

void draw()
{
  if(mySerial.available() > 0)
  {
    //set the value recieved as a String
    String value = mySerial.readString();
    //check to make sure there is a value
    if(value != null)
    {
      //add a new row for each value
      TableRow newRow = table.addRow();
      //place the new row and value under the "Data" column
      newRow.setString("Data", value);
    }
  }
}

void keyPressed()
{
  //save as a table in csv format(data/table - data folder name table)
  saveTable(table, "data/table.csv");
  exit();
}

*Make sure to adjust the code for the same COM port and and baud rate as your Arduino sketch*

This bit of code is great for one time data collection of your project, but it lacks a few things. In the next step, more code will be added to include date/time of each input and save the table with the date as the filename.

STEP 3: Add Timestamp to Data and File

This part will add code to the previous step and allow for files to be saved without having to change the save file name. The code will put a timestamp on each received value from the Arduino. When the Processing program is closed it will name the file with the timestamp(yyyy-mm-dd-hh-mm).

//import the required libraries
import processing.serial.*;

Serial mySerial;
Table table;
String filename;

void setup()
{
  //set mySerial to listen on COM port 10 at 9600 baud
  mySerial = new Serial(this, "COM10", 9600);
  
  table = new Table();
  //add a column header "Data" for the collected data
  table.addColumn("Data");
  //add a column header "Time" and "Date" for a timestamp to each data entry
  table.addColumn("Time");
  table.addColumn("Date");
}

void draw()
{
  //variables called each time a new data entry is received
  int d = day();
  int m = month();
  int y = year();
  int h = hour();
  int min = minute();
  int s = second();
  
  if(mySerial.available() > 0)
  {
    //set the value recieved as a String
    String value = mySerial.readString();
    //check to make sure there is a value
    if(value != null)
    {
      //add a new row for each value
      TableRow newRow = table.addRow();
      //place the new row and value under the "Data" column
      newRow.setString("Data", value);
      //place the new row and time under the "Time" column
      newRow.setString("Time", str(h) + ":" + str(min) + ":" + str(s));
      //place the new row and date under the "Date" column
      newRow.setString("Date", str(d) + "/" + str(m) + "/" + str(y));
    }
  }
}

void keyPressed()
{
  //variables used for the filename timestamp
  int d = day();
  int m = month();
  int h = hour();
  int min = minute();
  int s = second();
  //variable as string under the data folder set as (mm-dd--hh-min-s.csv)
  filename = "data/" + str(m) + "-" + str(d) + "--" + str(h) + "-" + str(min) + "-" + str(s) + ".csv";
  //save as a table in csv format(data/table - data folder name table)
  saveTable(table, filename);
  exit();
}

Things to note about the code:

-two new columns for time and date

-in void draw() (similar to void loop()) the time and date is called each time to ensure each variable is recorded at that time

-you can rearrange the columns an any order in void setup(), void draw() has no affect on the order of the columns

-the program when closed will save the file as .csv with the variable filename as the title. The title is a combination of the month, day, hours, minutes, and seconds.(This is to ensure that new files don't overwrite each other.)

*Code can be add to the the filename string to include the year*

NOTE: If there is other inputs that need to be in its own separate column, code would be needed to differentiate from each value received by Processing. This could be additional code for both the Arduino and Processing sketch.

Comments

[HectorT22]

Dear Henry, I want to generate file name and data & time in csv file in the format yyyymmdd hhmmss.zzz. Where: zzz is millisecond. e.g 20190709 090801.100.

I handled this task easily in C++ by using setfill and setw commands. But I haven't used Processing code before, so I don't know how to fill in the blank space, such as hour, minute less than 10 your code only shows 1 digit. Please tell me how to make the format with full digits.

Besides, after I modified your code a little and see the CSV file, all data are put in the same row with date & time. It should be each data corresponding a single second. I want it to update data every second for 30 sensor channels. How to make it work by Processing code?

Thanks in advance.

[parvinf]

Hi,

Thank you for the helpful tutorial. I run the code both in processing and Arduino without any problem, and when I press a key, it seems that the file has been saved, but, I cannot find the file. Can you help me with that?

[HenryC86]

If there is no error. Then it should be saved on the folder where the processing file is located. It will create new folder at that location named data and inside will be the csv file.

[AmanK113]

Hi , i did the same thing,but got an error in keypressed function while saving the table as xlsl file.

I am getting error- No extension specified for saving this table.

What should i do now?

[karlt1]

good tuto

take a look at :

e-mole.cz/diy/molegraph

and this app too

may be it will help you to have something better

[HenryC86]

Thanks! I'll take a look at it.