Introduction: Arduino String Manipulation Using Minimal Ram

By
About: Dr James Moxham is a general medical practitioner in Blackwood, Australia. His interests include general family medicine, medical politics, microcontrollers and tending a rose garden. He lives on a property wi…

An arduino Uno has 32k of flash memory but only 2k of ram. If we use a line of code like 

Serial.println("Hello World");

the text "Hello World" ends up being stored in ram, not in flash, and uses 11 bytes. Furthermore, if you start manipulating strings of text using the String class, the ram disappears rapidly. Even worse, if you use too many String class routines, the memory starts getting fragmented, and in extreme cases, runs perfectly for a week and then crashes. This project started out debugging a large program that was crashing after a week, and ended up solving several other problems, including debug messages that use no ram and string routines that do not use the String class and conserve as much ram as possible. This is an excellent article on the perils of the arduino String class

https://hackingmajenkoblog.wordpress.com/2016/02/04/the-evils-of-arduino-strings/

Let's start with a completely blank program on a Uno. This uses 9 of the 2048 bytes of ram. Add Serial.begin() and it uses 182 bytes. Add Serial.println(""); and it uses 200 bytes, and we still haven't actually done anything useful yet! Add Serial.println("0123456789"); and this uses 210 bytes. Now, there is a nifty hack which moves that last line of text into flash memory - the F macro. So Serial.println(F("0123456789")); and it goes back to using 200 bytes. However, there are two reasons we can't use this. First is that it only works with Serial - so if you want to output to an LCD display it won't work. And second, it really is only useful for printing messages to the screen. You can't use the F macro to store a string which you then want to manipulate later. Fortunately, we can use PROGMEM instead to store text in flash.

To start with, let's define three string arrays - two input arrays and one output array. These are each a fixed 80 bytes long which should be plenty, and together use 240 bytes of ram. When compiled and with all the other code that gets added, the amount of ram being used is 422 bytes. There will be some stack space used each time a function is called for local variables and arrays, but this is returned at the end of the function. If no more than three or four temporary arrays exist in a function, and each is a maximum of 80 bytes, there will be no chance of the stack memory and the heap memory growing and overwriting each other and causing a crash. The global arrays are 

char inputString1[80]; // general purpose input string, 80 bytes 
char inputString2[80]; // general purpose input string, 80 bytes 
char outputString[80]; // general output string, 80 bytes

So, we want to print out "Hello World", and we want to store that text in flash memory rather than in ram. Whatever code we add, the ram used must stay at 422 bytes. There are several steps to this - store the text in flash. Move it to a temporary array which only exists in the local function, then move it from that array to the global inputstring array above. Then print it. Seems a bit complicated, and yes it is as even a debug message now needs two lines of code. The upside - you can do repeat this over and over and not use any more ram. 

const static char s1[] PROGMEM = "Hello World"; 
printStringln(getString(s1));

Next step - moving strings from flash to one of our global string arrays so we can start manipulating those strings 

const static char s1[] PROGMEM = "Long string of text"; 
strcpy_P(inputString1, s1); // copy to one of the temp strings

Now we have the string in the general purpose inputString array, we can do things to it. Find a letter in the string. Chop it up. Combine it with another string. In a generic sense, two input strings and one output string seems to be enough to do most things, even for reading in pages of text a line at a time from an SD card and manipulating the text. In the attached arduino program are more examples. This little function returns the left characters. Pass it "Hello World" and 5, and it returns "Hello". 

char* basicLeftString(const char *str, int i) 
{
    strncpy(outputString, (char*)str, i); // copy to outputString
    outputString[i] = '\0'; // put in new null terminator
    return outputString;
}

One other thing mentioned above - porting code between different displays. I have come across this problem writing a program using lots of Serial.println, and then later changing to an LCD display. Lots of lines of code to change, so I figured that if the output to the serial port only existed in one place in the program, it would only be one line to change. Everything that is displayed goes through this one routine: 

void printString(const char *str) 
{ 
    const char *p;
    p = str;
    while (*p) {
        Serial.print(*p); // explanation in majenko's webpage
        p++;
    }
}

So now for some string routines that replicate the String class. Add strings together, cut them up, compare them, convert them to numbers, convert numbers to strings that are decimal, binary, hex. Fortunately, good old fashioned C has these routines and they all work fine on the arduino. Things like strcmp(), strcpy(), strcat() atoi(). The complicated part is coding in a way that doesn't use ram, and sometimes the only way to do that is to add lines of code one at a time and keep compiling and checking the free ram number. Oh, and yes, there are Pointers. It is quite possible to code the arduino never having to use a pointer, but unfortunately, they are needed when you abandon the String class. Pointers are just a number that shows where the start of a string is in memory, but they do involve using the little * character in ways that seem rather obtuse. Sometimes the * is at the start of a word, sometimes at the end, sometimes there are brackets, sometimes not. Hopefully the attached code has enough examples that are repeated over and over to show what the pattern is. And there is also Type Casting - which is needed to tell the compiler explicitly what type of variable is being used. In really old fashioned Basic, you might have added two strings together with A$ = B$ + C$. Using "no String class" C, it looks more like 

char* outputString = stringAdd(inputString1,inputString2);

which calls a function:

char* stringAdd(const char *str1, const char *str2) 
{
    strcat((char*) str1,(char*) str2); // add strings together, answer in str1
    strcpy(outputString, str1); // copy string to outputstring
    return outputString;
}

Demonstration routines below. Searching on the internet brings up many more useful routines using plain old vanilla C. 

void setup() {
   
 Serial.begin(9600);    // start serial port
    messageTestExample();  // print Hello World
    parseTest();           // create a string in flash, move to 
inputstring, move to outputstring, print on the screen
    printNumber(19);       // print a number on the screen
    stringCompareTest();   // create two strings, compare if equal. Also
 shows how debug messages are in flash, saving precious ram
    integerToStringTest(); // integer to a string
    stringToIntegerTest(); // string to an ingeter
    longToStringTest();    // long to a string 1234
    longToBinaryTest();    // long to a binary string 110101
    longToHexTest();       // long to a hex string 123ABC
    // old Basic functions
    leftStringTest();      // get the left characters from a string
    midStringTest();       // get the middle characters from a string
    instrTest();           // find where a string is in another string
    lenTest();             // length of a string
    ascTest();             // character to a number
    chrTest();             // number to a character
    hexTest();             // number to a hex string
    basicStrTest();        // number to a string
    basicValTest();        // string to a number
    basicStringAddTest();  // add two strings together
    finishMessage();       // for debugging, string errors tend to 
corrupt code so it never gets to here, so make sure this prints
}

I hope someone finds this useful. Thoughts and suggestions would be most appreciated as I'm sure there are better ways to do this!

Step 1: Code


char inputString1[80]; // general purpose input string, 80 bytes
char inputString2[80]; // general purpose input string, 80 bytes
char outputString[80]; // general output string, 80 bytes
void setup() {
    Serial.begin(9600);    // start serial port
    messageTestExample();  // print Hello World
    parseTest();           // create a string in flash, move to inputstring, move to outputstring, print on the screen
    printNumber(19);       // print a number on the screen
    stringCompareTest();   // create two strings, compare if equal. Also shows how debug messages are in flash, saving precious ram
    integerToStringTest(); // integer to a string
    stringToIntegerTest(); // string to an ingeter
    longToStringTest();    // long to a string 1234
    longToBinaryTest();    // long to a binary string 110101
    longToHexTest();       // long to a hex string 123ABC
    // old Basic functions
    leftStringTest();      // get the left characters from a string
    midStringTest();       // get the middle characters from a string
    instrTest();           // find where a string is in another string
    lenTest();             // length of a string
    ascTest();             // character to a number
    chrTest();             // number to a character
    hexTest();             // number to a hex string
    basicStrTest();        // number to a string
    basicValTest();        // string to a number
    basicStringAddTest();  // add two strings together
    finishMessage();       // for debugging, string errors tend to corrupt code so it never gets to here, so make sure this prints
}
void loop() 
{
}
//******************* Test examples *********************
void messageTestExample()
{
  const static char s1[] PROGMEM = "Hello World"; // store the text in flash
  printStringln(getString(s1));
}
void parseTest() 
{
  const static char s1[] PROGMEM = "Parse Test"; // move fixed text into temporary array
  strcpy_P(inputString1, s1); // copy to one of the temp strings, must use strcpy_P not strcpy
  char* outputString = parseString(inputString1); // move to outputString, note the char* before outputString
  printStringln(outputString); // print out
}
void stringCompareTest() // compare two strings
{
  uint8_t a; // a byte
  const static char s1[] PROGMEM = "Long string of text"; // move fixed text into temporary array
  strcpy_P(inputString1, s1); // copy to one of the temp strings, must use strcpy_P not strcpy
  const static char s2[] PROGMEM = "Long string of text"; // move fixed text into temporary array
  strcpy_P(inputString2, s2); // copy to the other input string
  a = stringCompare(inputString1,inputString2); // 0 is a match, uses strcmp(str1, str2);
  if (a == 0) {
    const static char s3[] PROGMEM = "String compare - Strings match"; // store the text in flash
    printStringln(getString(s3));
  }else{
    const static char s4[] PROGMEM = "String compare - Strings do not match"; // store the text in flash
    printStringln(getString(s4)); 
  }
}
void leftStringTest()
{
  const static char s1[] PROGMEM = "Test left string routine"; // move fixed text into temporary array
  strcpy_P(inputString1, s1); // copy to inputString1
  char* outputString = basicLeftString(inputString1, 14); // left most characters, 1 returns one character, 0 not allowed
  printStringln(outputString); 
}
void midStringTest()
{
  const static char s1[] PROGMEM = "Test mid string routine"; // move fixed text into temporary array
  strcpy_P(inputString1, s1); // copy to inputString1
  char* outputString = basicMidString(inputString1, 6, 10); // mid characters, first is 1, start at 6, and return 10 characters
  printStringln(outputString); 
}
void stringToIntegerTest() 
{
  int i;
  const static char s1[] PROGMEM = "500"; 
  strcpy_P(inputString1, s1); 
  i = stringToInteger(inputString1); // value is in i
  i = i + 100; // do something to this number
  printNumber(i);
}
void longToStringTest()
{
    char* outputString = longToString(-12345678); // long to string
    printStringln(outputString);
}
void longToBinaryTest()
{
    char* outputString = longToBinaryString(65535);
    printStringln(outputString);
}
void longToHexTest()
{
    char* outputString = longToHexString(65535);
    printStringln(outputString);
}
void integerToStringTest()
{
    char* outputString = integerToString(1234); // convert integer to a string
    printStringln(outputString);
}  
void instrTest() // uses strstr, see also strchr to find a character in a string
{
  int index;
  const static char s1[] PROGMEM = "Find a needle in a haystack"; 
  strcpy_P(inputString1, s1); 
  const static char s2[] PROGMEM = "needle"; 
  strcpy_P(inputString2, s2); 
  index = basicInstr(inputString1,inputString2); // find where needle is
  printNumber(index);
}
void lenTest() 
{
  int i;
  const static char s1[] PROGMEM = "Hello World"; 
  strcpy_P(inputString1, s1); 
  i = basicLen(inputString1);
  printNumber(i);
}
void ascTest()
{
  uint8_t n;
  const static char s1[] PROGMEM = "A Hello World"; // should print 65
  strcpy_P(inputString1, s1); 
  n = basicAsc(inputString1);
  printNumber(n);
}
void chrTest()
{
  char* outputString = basicChr(66); // ascii B
  printStringln(outputString);
}
void hexTest()
{
  char* outputString = basicHex(254);
  printStringln(outputString);
}
void basicStrTest() // number to string
{
    char* outputString = basicStr(120); // same as long to string
    printStringln(outputString);
}
void basicStringAddTest()
{
  const static char s1[] PROGMEM = "String "; // move fixed text into temporary array
  strcpy_P(inputString1, s1); // copy to one of the temp strings, must use strcpy_P not strcpy
  const static char s2[] PROGMEM = "Add"; // move fixed text into temporary array
  strcpy_P(inputString2, s2); // copy to the other input string
  char* outputString = basicStringAdd(inputString1,inputString2);
  printStringln(outputString);
}
void finishMessage() 
{
 const static char s1[] PROGMEM = "Finished"; // print a finish message, bugs tend to corrupt the final message so end with this
 printString(getString(s1));
}
//******************** String Routines ******************
char* getString(const char* str) // String replacement - move string from flash to local buffer
{
  strcpy_P(outputString, (char*)str);
  return outputString;
}
void printString(const char *str) // all output directed through this one routine, so can change Serial.print to whatever display is being used
{ 
    const char *p;
    p = str;
    while (*p) {
        Serial.print(*p); // explanation in majenko's webpage
        p++;
    }
}
void printStringln(const char *str) // print line with crlf
{
    printString(str); 
    crlf(); // carriage return, line feed
}
void crlf() // carriage return and linefeed
{
  const static char crlf[] PROGMEM = "\r\n"; // carriage return, then line feed, maybe not the most efficient way to do this but works
  printString(getString(crlf)); // print out, using one central function for output so easier to change destination with different displays
}
void printNumber(long n)
{
  char* outputString = integerToString(n); // convert to number
  printStringln(outputString);
}
int stringCompare(const char *str1, const char *str2) 
{ 
  uint8_t a;
  a = strcmp(str1, str2); // 0 is a match
  return a;
}
char* basicLeftString(const char *str, int i) 
{
    strncpy(outputString, (char*)str, i); // copy to outputString
    outputString[i] = '\0'; // put in new null terminator
    return outputString;
}
char* basicMidString(const char *str, int stringStart, int stringLength) 
{
    strncpy(outputString, (char*)str + stringStart - 1, stringLength); // copy to outputString using Basic nomenclature where 1 is the first character
    outputString[stringLength] = '\0'; // put in new null terminator
    return outputString;
}
char* integerToString(int n) // returns outputString
{
  itoa(n, outputString, 10); // itoa is for integers, 10 is for base 10 (could use 2 for binary, 16 for hex)
  return outputString;
}
char* longToString(long n)
{
  ltoa(n, outputString, 10); // base 10
  return outputString;
}
char* longToBinaryString(long n)
{
  ltoa(n, outputString, 2); // base 2 is binary
  return outputString;
}
char* longToHexString(long n)
{
  ltoa(n, outputString, 16); // base 16 is binary, returns lower case A-F
  return outputString;
}
int stringToInteger(const char *str1) 
{
  int i;
  i = atoi(inputString1);
  return i;
}
char* parseString(const char *str1) 
{
  strcpy(outputString, str1); // copy string
  return outputString;
}
int basicInstr(const char *haystack, const char *needle) // same as Basic instr
{
  char *e;
  int index;
  e = strstr(haystack, needle); // get pointer to the string. See also strchr which looks for just a single character
  index = (int) (e - inputString1); // find where the substring is
  index = index + 1; // add one so same as Basic
  return index;
}
int basicLen(const char *str1)
{
  int i;
  i = strlen(str1);
  return i;
}
uint8_t basicAsc(const char *str1) // returns ascii value of the left most character in a string. 
{
  uint8_t c;
  c = str1[0]; // get the value
  return c;
}
char* basicChr(uint8_t c) // convert c to a string of length 1 character
{
  outputString[0] = c;
  outputString[1] = '\0'; // null terminator
  return outputString;
}
char* basicHex(uint32_t n) // can be any sort of number, long, byte, uint8,16,32 etc
{
  ltoa(n, outputString, 16); // base 16 is binary, returns lower case A-F
  return outputString;
}
char* basicStr(long n)
{
  ltoa(n, outputString, 10); // base 10
  return outputString;
}
long basicVal(const char *str1) 
{
  long i;
  i = atol(inputString1); // returns a long but can cast into other types
  return i;
}
void basicValTest() 
{
  long i;
  const static char s1[] PROGMEM = "4567"; 
  strcpy_P(inputString1, s1); 
  i = basicVal(inputString1); // value is in i
  printNumber(i);
}
char* basicStringAdd(const char *str1, const char *str2) // adds string 2 to the end of string 1
{
    strcat((char*) str1,(char*) str2); // add strings together, answer in str1
    strcpy(outputString, str1); // copy string to outputstring
    return outputString;
}

Attachments