Introduction: Remote High Speed Data Logging Using Arduino/GL AR150/Android/pfodApp

This instructable shows you how to remotely log and plot data at up to ~2k (2000) samples/sec using a local network formed from an Adafruit WICED, a GL AR150 mini 5V router and an Android mobile running pfodApp.(V2.0.224+). Since all these devices run from 5V, this set up is suitable for field and lab work. These instructions are also available here.

First let's define what we mean by High Speed Data Logging. For this tutorial, it means a sustained data rate of up to ~40K bytes/sec. That is, being able to sample at ~2000 samples/sec and send the time stamped measurement to a remote Android running pfodApp for logging and plotting with each measurement being approximately 20 bytes (depending on the number of bytes in the time stamp and sample data). While higher burst sampling rates are possible by relying on the internal and network buffers, this page only considers sustained real time sampling rates.

First we will discuss the bottlenecks that restrict the data rate and presents a series of simple test programs to help you quantify them. When using pfodApp to log and plot the data, you can turn on the debug logging to record the rate at which the app is processing the data. That rate varies between mobiles and Android versions and also varies depending of what other apps are running.

The tests here show that using an Adafruit WICED Feather sending data via a GL AR150 mini router to a ASUS zenfone5,running pfodApp (V2.0.224+), you can log an plot at ~40Kb/sec. On a Nexus P6 you can log and plot at ~48K bytes/sec. If you have a laptop handy you can use TeraTerm or CoolTerm to log the data directly to it and so avoiding using pfodApp. However pfodApp/pfodDesignerV2 makes it easy to control other functions on your Arduino as well as collecting and plotting the data.

Step 1: Bottlenecks to High Speed Data

There are four (4) possible bottlenecks in logging/plotting high speed data :-

  1. Speed of sampling the data in the Arduino
  2. Speed of transferring the data to the communication shield or sub-system
  3. Speed of the communication channel
  4. Speed of the receiving Android device.

Looking at these in turn

Step 2: 1. Speed of Sampling the Data in the Arduino

The sampling rate in the Arduino needs to be faster then the desired samples/sec because the Arduino needs to do other work, like scaling and handling the communications, as well as doing the sampling. Sometimes the measurement hardware has sever sample/sec limitations or the protocol (SPI, Serial, I2C, etc) used to connect the measurement hardware to the Ardunio limits the maximum samples/sec.

This example sketch, SamplingSpeedTest_Serial.ino, lets you test the maximum samples / sec for your measurement hardware. It uses analogRead() but you should replace the line of code with what ever code you need to get the sample. You should also add any extra code you need to scale/normalize the reading and prepare the measurement time stamp. (If you find the scaling calculations are taking too long, try removing all floating point calculations and send the data as integers.)

The SampleSpeedTest_Serial.ino is initially set to work with Arduino boards and to send the data to a NullPrint. Here are some results for typical boards.

Aduino Mega/Uno
Using NullPrint 1000 loops in:839mS-> Sample rate:1.19k samples/sec
Example data record:

Arduino 101
Using NullPrint 1000 loops in:83mS-> Sample rate:12.05k samples/sec
Example data record:

To test the Adafruit WICED you need to uncomment the
#define USE_WICED
line at the top of the SampleSpeedTest_Serial.ino sketch. WICED has non-standard pin names and pinModes (and some non-standard function names also). For the WICED board the results are higher. WICED also has an ADC higher speed burst mode using an internal buffer, but that is not used here.

Adafruit WICED
Using NullPrint 1000 loops in:71mS -> Sample rate:14.08k samples/sec
Example data record:

Step 3: 2. Speed of Transferring the Data to the Communication Shield or Sub-system

If your board does not include an integrated communication system you need to either send the data to the Serial port (USB) or transfer the data to a Wifi/Ethernet/Bluetooth shield to send it remotely. The speed of to transfer to the communication shield or sub-system is the next possible bottleneck.

Serial connections

Using Serial, each byte of data is transmitted as 10 bits, 1 start, 8 data and 1 stop bit. So a 115200 baud Serial connection can transfer about 115200/10 = 11520 bytes/sec. If there are 14 bytes in each data record (as there are in these tests), that implies to transmit one data record at 115200baud takes ~1.2mS or ~1215mS for 1000 records. (or 14580mS at 9600baud)

To test the USB serial transfer rate, in SampleSpeedTest_Serial.ino, set
boolean useNullPrint = false;
and set the baudrate
const unsigned long baudRate = 115200;

Here is the result for Arduino Mega and Uno at 9600baud

Arduino Mega/Uno
Using Serial at 9600 1000 loops in:15600mS -> Sample rate:0.06kHz
Example data record:

and at 115200baud

Arduino Mega/Uno
Using Serial at 115200 1000 loops in:1275mS -> Sample rate:0.77kHz
Example data record:

Here is the result for the Arduino 101 at 115200baud

Arduino 101
Using Serial at 115200 1000 loops in:1142mS -> Sample rate:0.88kHz
Example data record:

To test the Adafruit WICED you again need to uncomment the
#define USE_WICED
line at the top of the SampleSpeedTest_Serial.ino sketch.

The Adafruit WICED has a high speed USB serial connection which ignores the baud rate setting and just sends as fast as it can, ~60K bytes/sec

Adafruit WICED
Using Serial at 115200 1000 loops in:235mS -> Sample rate:4.26k samples/sec
Example data record:

So if you can connect directly to your computer via USB then WICED can log data at about 60K bytes/sec directly.If you instead send the WICED data to a standard serial port, e.g. Serial1, then the results are the same as the Arduino boards.

Arduino Ethernet Shield V1 SPI connection.

The sketch, SamplingSpeedTest_Ethernet.ino, tests the Ethernet Shield sample rate, connection to TeraTerm running on computer connected to the same network

The Arduino Ethernet Shield V1 SPI connection/shield is very slow.
Using Ethernet Shield V1 1000 loops in:2877mS -> Sample rate:0.35kHz
Example data record:

It is not clear if this slowness is due to a) slow SPI speed or b) slow TCP/IP handling on the Ethernet shield. Adding buffering (see below) did not significantly improve the result. This shield has been replaced by a newer Arduino Ethernet Shield V3 but that shield has not been tested.

Direct Connections

Some boards provide direct transfer of the data to an on-board communication module. Examples of these boards are Adafruit WICED, ESP8266, ESP32, Arduino 101, etc. This on-board transfer is difficult to measure but is usually not the limiting factor.

Step 4: 3. Speed of the Communication Channel

The next bottleneck is the remote transfer of the data from the Arduino communication module to the remote Android mobile or computer. For Android the following options are available, slowest to fastest, SMS, Bluetooth Low Energy, Bluetooth (Classic), WiFi/Ethernet (TCP/IP). For computers generally only Bluetooth (Classic) and WiFi/Ethernet (TCP/IP) are available.

Since we are considering high speed data logging, only TCP/IP will be considered. As noted above, the Arduino Ethernet V1 shields are slow and so Ethernet will not be considered further here. (Arduino Ethernet Shield V3 has not been tested). This leaves WiFi as the communication channel for high speed data logging.

Not all WiFi (TCP/IP) implementations have the same speed. In particular the WiFi built into the ESP8266 can only send one packet at a time and waits for that packet to be ACKed before sending the next one. This severely limits its through put. The ESP32 may be better but the Arduino support is currently (as of 4th December 2015) incomplete. The Adafruit WICED board has a better TCP/IP implementation but even with this board, the default code will send lots of small packets. The sketch, WICED_simpleWifiSpeed.ino, finds the speed using the default code with small packets.

Using (unbuffered) Adafruit WICED 1000 loops in:5031mS -> Sample rate:0.20k samples/sec
Example data record:

Lots a small TCP/IP packets severely limits the through put. Adding buffering to the output, so that the data is held until there is a (almost) full TCP/IP packet and then sending it, gives a much faster through put. The buffered files are in this zip file.

Using buffered Adafruit WICED 1000000 loops in:131509mS -> Sample rate:7604.04 samples/sec
Time left to completion: 0.00 sec
Example data record:

The buffering class used here is pfodBufferedClient and is it is zipped up with the revised sketch, WICED_bufferedWifiSpeed.ino, in this zip file. Adafruit WICED has a client.usePacketBuffering(true) method, that also does buffering. Unfortunately it will wait for ever to send a partial buffer if you don't call flush(). The pfodBufferedClient class has a short timeout which sends the packet if nothing has been added in the last 10mS. This timeout is checked on each call to read() available(), peek() or stop() and so is more convenient to use. Adding this buffering to the Arduino Ethernet Shield V1 did not significantly increase its through put.

So to summarize so far, of the various boards I have to hand, the Adafruit WICED is the fastest. Over a 20min test it sent about about ~120K bytes/sec to TeraTerm running on my computer. If you have a computer handy on the WiFi network then you can use TeraTerm or CoolTerm to capture the data to a file. On the other hand if you are using pfodApp to control the board and capture and chart the samples, the data rate also depends on the processing speed of your Android device and its TCP/IP buffers.

Step 5: 4. Speed of the Receiving Android Device

The final bottleneck in high speed data logging is the receiving device, be it a computer or a mobile. The previous section tested the speed connecting to a computer. For testing connecting to an Android mobile, pfodApp will be used. For 'remote' data logging a small portable wifi router is used, GL AR150.

To test the Android mobile's speed, the free pfodDesignerV2 app was used to design a simple menu with just one button which opens a single plot of the sampled analog data. Adafruit WICED and pfodDesignerV2 covers designing a menu for WICED and Adding a Chart and Logging Data on How to Display/Plot Arduino Data on Android has details on setting up a chart with one plot and logging data.

The resulting sketch generated by pfodDesignerV2 for the WICED target is WICED_chart.ino and produces the following menu and chart. Data samples are only sent at 1/sec.

The WICED_chart.ino sketch was then modified to add buffering and to send the data at a much faster rate. The files use for the following tests are in Unzip those to your Arduino directory and compile and load the WICED board with the pfodWICED_highSpeedChart sketch.

Install pfodApp and set up a connection to your WICED board as described in pfodAppForAndroidGettingStarted.pdf. To see in more detail what is happening, in pfodApp open Add/Edit for the WICED connection and tick the Log Debug data tick box. This will log to the WICED_debug.txt file under /pfodAppDebugData directory on your mobile. By default, Log Plot Data is ticked and pfodApp saves any data received to a file on your Android device under the /pfodAppRawData directory.

After you exit pfodApp (which flushes the last buffer), transfer the /pfodAppDebugData/WICED_debug.txt file to you computer (using say the wifi file transfer app) and open it in a text editor. You will find lines like

< 1481329581822 4.962sec dataRate:42Kbytes/sec { }
> 1481329581860 0.038sec dataRate:42Kbytes/sec {}

In this instance pfodApp was running on an ASUS zenfone5 mobile, running Android 4.4.2. The dataRate: is the Kbytes/sec since the last pfod message. The data rate varies between mobiles and Android versions and also varies depending of what other apps are running.

The < … { } is a keep alive message sent very 5sec if no other pfod commands are being sent by the user. The > … {} is the response back from the WICED board 0.038sec later. If the data rate is too high for the Android mobile to handle then the 0.038sec will progressively get longer and longer and eventually exceed the connection timeout (default 10sec) and cause the pfodApp to disconnect and reconnect.

The pfodApp chart is updated about once per sec. You can freeze the plot by putting your finger on the chart. You can also use two fingers to zoom the chart and drag the chart.

Reducing the PLOT_DATA_INTERVAL to 500uS

unsigned long PLOT_DATA_INTERVAL = 500;// uS == 0.5mS == 2kHz samples , edit this to change the plot data interval

doubles the sample rate so that the data rate is about 40Kbytes/sec.

The ASUS zenphone5 running Android 4.4.2 has a processing dataRate is about 40Kb/sec, if the WICED's data rate exceeds that, pfodApp cannot keep up and quickly times out. You can set the pfodApp connection time out to 0 so it never times out, but eventually the WiFi TCP buffers will fill up and block the data being sent.

When running pfodApp on a Nexus P6 you can reduce the PLOT_DATA_INTERVAL to 400uS

unsigned long PLOT_DATA_INTERVAL = 400;// uS == 0.4mS == 2.5kHz samples , edit this to change the plot data interval

which give a data rate of about 48Kb/sec.


There are a number of factors that limit the data rate for an Arduino/Android based system. Of the boards I have here, the Adafruit WICED connected via WiFi to a ASUS zenfone5 or Nexus P6 running pfodApp (V2.0.224+), was the fastest, achieving ~40Kb/sec (~2000 samples/sec.) The pfodApp can also plot the data and you can freeze the plot by putting you finger on it. Using a 5V mini router, like the GL AR150, allows you to take the whole set up out in the field.

Arduino Contest 2016

Participated in the
Arduino Contest 2016