Introduction: Write Data to Beginning of File With Arduino or ESP8266

About: IoT - Internet of Things. Iota - small thing. Thingamajig - An object whose name can't be recalled. Iotamajig - A little unnamed internet connected gizmo!

I had a data logging project where I needed to log data to a file, then once a day, send the data file to a central server. I was using an ESP8266-01 to do this, leveraging the Arduino IDE.

This is easy enough to do using the SPIFFS/File System Object for Arduino, which is very well documented here on github.

The problem that I had, was that I needed to post data to the beginning of the file. The file system object in Arduino makes it very easy to append new lines at the end of a file, but I didn't see anything that would shove all the existing data down a row and put new data at the beginning.

The reason I wanted to put the data at the beginning was because I needed to match another file generated by another system that I can't modify code for.

I hope that there's an easier way to do this that I overlooked, but I couldn't find anything. So this is what I came up with:

Step 1: The Code

#include "FS.h"

int firstRun = 1;

unsigned long prvTime = 0;
long intLog = 5000;  // Log every 5 seconds
float datArray[100]; // our data array holding 100 elements

unsigned long readTime = 0;
long intRead = 60000;  // Read every 60 seconds

void logData(unsigned long newData){

    Serial.print("Logging: ");
    Serial.println(newData);

    for (int i = 99; i >= 0; i--) {

      if (i == 0) {
        datArray[i] = newData;
      } else {
        datArray[i] = datArray[i - 1];
      }

    }

    //open file for appending new blank line to EOF.
    File f = SPIFFS.open("/data.txt", "w");

    for (int i = 0; i < 100; i++) {
      f.println(datArray[i]);
    }
    f.close();
    
}

void readData(){
  
  int xCnt = 0;
  
  File f = SPIFFS.open("/data.txt", "r");
  
  if (!f) {
      Serial.println("file open failed");
  }  Serial.println("====== Reading from SPIFFS file =======");

    while(f.available()) {
      //Lets read line by line from the file
      String line = f.readStringUntil('\n');
      Serial.print(xCnt);
      Serial.print("  ");
      Serial.println(line);
      xCnt ++;
    }
    f.close();    

}

void setup() {

  int xCnt = 0;

  Serial.begin(115200);

  //if a temp file has already been in use, open it and load it's data to array.
  SPIFFS.begin();

    File f = SPIFFS.open("/data.txt", "r");

    if (!f) {
      Serial.println("File open failed");
    } else {

      //we're only tracking the first 100 lines, so we don't care if there is anything after that.
      while(f.available() && xCnt < 100) {
        //Lets read line by line from the file
        String line = f.readStringUntil('\n');
        datArray[xCnt] = line.toInt();
        Serial.print(xCnt);
        Serial.print("  ");
        Serial.println(line);
        xCnt ++;
      } 
    }
    f.close();

}

void loop() {
  
  unsigned long curTime = millis();

    if (firstRun == 1 ) {
      firstRun = 0;
      prvTime = curTime;
      logData(prvTime); 
      readTime = curTime;   
    }
  
    if (curTime - prvTime > intLog) {
      prvTime = curTime;
      logData(prvTime);
    }

     if (curTime - readTime > intRead) {
      readTime = curTime;
      readData();
    }

}

Step 2: Explanation

In this example, we are going to log a piece of data every 5 seconds, and then print the file to Serial once a minute.

This is just for demonstration, and simplified to the basics so you can easily get it up and running. My real project tracks data once every 30 minutes and uploads the results once a day.

What I DON'T like about this code, is that I'm keeping double track of all the data: once in an array, and once in the file. The easiest way I found to shift all the data down a line in the file, was to repopulate the file each time I add a record, by rereading it from the array.

Why use a file you ask? The array already has the data... Well, if the ESP loses power, the array loses its data. So by writing to the file, I never lose the data. When the ESP starts, it reads the file back in to the array, and everything continues on its merry way!

This has been a very stable solution to me, tracking just a few pieces of data at a time. It's run for months with no issues. If you have a more complex project, this method might run into issues depending on how many types of data you want to collect and for how many intervals... but I say, hey, use it till you break it!

If you need a way to post data at the beginning of a file while programming with the Arduino IDE, then hopefully this has helped. Thanks for reading.