Introduction: High Speed ESP32 Control for Beginners

By pfodApps and pfodDevices

This tutorial shows you how to use the HP_AsyncTCP library to run your ESP32 Arduino loop() as fast as you can without the user control and data logging slowing it down. This lets you consistently detect and respond to external changes within a few 10's of micro seconds. The examples here run a stepper motor from an ESP32 loop() that executes at least every 40us while still logging data and allowing user control.

This is achieved by configuring the ESP32 to allocate one core (core 1) solely to running the Arduino loop() at the highest speed possible and deferring all the control, commands and data logging to the async... methods that run on the second core ((core 0) and connect to a TCP client via WiFi.

This is suitable “for beginners” because it only extra things it needs over the 'basic' Arduino/ESP32 examples is the use of the volatile keyword and the replacement of any delays with the non-blocking millisDelay.

Note there are other ESP32xx versions like the ESP32-S2 and ESP32-C3 etc that only have a single core. This page is only for the dual core ESP32 chips (ESP32 / ESP32-S3). The HP_AsyncTCP library was developed for the Europa Prototype Ice-Melting Probe for Collecting Biological Samples proof of concept system.

What about using Interrupts for high speed control.

Interrupts let you detect pin changes quickly but the interrupt code needs to be short, so for non-trivial actions you still need to loop() code run and see and act on the results of the interrupt routine. Using the techniques from Simple Multitasking Arduino you can often avoid using Interrupts by just polling the pin state directly from the loop() code. So unless you are trying to detect fleeting pin transitions that are only there for <50us, polling the pin from a fast loop() is simpler and just as effective.

The examples

The first example is of a trivial loop that shows how to log data without slowing down the loop and how even minimal logging slows the loop() significantly. The second example extends on that to both log data and control a stepper motor at speeds up to 20,000 step/sec. The third example shows how to use pfodApp to control the stepper and plot and log the data without slowing down the loop().

This tutorial is also available online

Supplies

Parts List (as at 1st Dec 2023 excluding shipping)

Basic

Sparkfun ESP32 Redboard ~US$30

ESP32 Arduino package V2.0.11 install via Arduino Boards Manager

HS_AsyncTCP library V1.0.0 installed from it zip file This is a modification of the AsyncTCP library

SafeString library V4.1.30 installed via Arduino Library Manager, for the millisDelay class

SpeedStepper library installed from zip file

The example sketches HighSpeedESP32_ex1.zip and HighSpeedESP32_ex2.zip

For convenience those three libraries are in this libraries.zip file. Rename you existing Arduino libraries dir to libraries_old and unzip that you can unzip this libraries.zip file to your Arduino sketch directory to get the new libraries directory

Optional

pfodDesignerV3, free Android app for designing pfodApp menus and charts for controlling your project

pfodApp ~US$12 Android app to display those menus and charts and to log the data.

pfodParser library V3.62.0 installed via Arduino Library Manager, to parse the pfodApp cmds

Optionalfor designing your own GUI controls

pfodGUIdesigner, free Android app for designing pfodApp gui screens

Step 1: A Simple Example, HighSpeedESP32_ex1

This first simple example covers how to code the asyncLoop() and how to read data from your Arduino loop() for logging via a telnet connection. It illustrates how even minimal logging to the Serial Monitor increases the maximum the loop() run time by more then 6 times. This means every now and a again your code will be 6 times slower to detect and respond to external changes.

Unzip the HighSpeedESP32_ex1.zip to your Arduino sketch directory. Fill in you network SSID and password in the HighSpeedESP32_ex1.ino file. Edit the staticIP to one suitable for your network. Select your ESP32 board, here it is a SparkFun ESP32 Redboard, and program it. The serial monitor, at 115200, will show when it has connected to your network.

Running with Telnet

The open a telnet connection (port 23) to the static IP you set. On Mac you can use the terminal (Command-N). On Windows you can use TeraTerm. Here TeraTerm is being used on a Window 10 machine. Sample telnet output is :-

Results output every 2sec.
millis, avg us/loop, max us/loop
92047,8.01,5669929
94047,8.04,25
96047,8.04,25
98047,8.04,25
100047,8.04,25

This show that the maximum to run this trivial loop(), which does not send and data to the Serial Monitor, is ~25us.

Now edit the HighSpeedESP32_ex1.ino (near the top) to set the PRINT_DELAY_MS to 1000 (i.e. 1sec) and run again. This time there is output on the Serial Monitor and the telnet output is:-

Results output every 2sec.
millis, avg us/loop, max us/loop
12241,9.02,5721952
14241,9.02,159
16241,9.02,162
18241,9.02,160

How Fast does the loop() run?

For the trivial loop() code in ex1, when not printing to the Serial Monitor, the average loop() time is ~8us and the maximum is ~25us. When the millis() and avg us/loop is output to the Serial Monitor at 1sec intervals the maximum loop() execution time increases by more than 6 times to ~160us

Even a small output to the Serial Monitor at a modest rate causes a significant delay in the loop()

So for consistent high speed loop control don't use Serial I/O

Step 2: How to Code for Telnet Data Logging and Control

HighSpeedESP32_ex1 and HighSpeedESP32_ex2 each consist of the usual Arduino .ino file and two support files. The WiFiDataHandling.cpp / .h files implement the async... methods that handle the WiFi data logging and control. The VolatileVars.cpp / .h files define the volatile variables that pass information between the Arduino loop() running on core 1 and the asyncLoop() running on core 0.

For code clarity each volatile variable is suffixed with _v, but it is not required. The file name VolatileVars is also not mandatory. It can be called what ever you like.

The WiFiDataHandling file implements the 5 methods that the HS_AsyncTCP library expects to find in your code. The name WiFiDataHandling is not special. Any file can contain these methods. But because all these methods are run on core 0 (the WiFi core), they should normally be in a separate .cpp file from your Arduino code that will run on core 1. Putting them in a separate .cpp file will help ensure that the interactions between the Arduino loop() and the async... methods are only via the volatile variables.

  void asyncSetup();
  void asyncLoop(Stream &stream);
  void asyncConnected(Stream &stream);
  void asyncDisconnected();
  void asyncDataReceived(Stream &stream);


Any of these methods can just be empty if you don't need them. asyncSetup() and asyncLoop() are the core 0, WiFi equivalents of the Arduino setup() and loop(). The other three methods are called when a client connects, disconnects or sends commands to your Arduino board.

In ex1, asyncSetup() starts the WiFi datalogging timer and initializes the async variables

void asyncSetup() {
  dataTimer.start(DATA_TIMER_MS);
  microsStart = micros();
  lastLoopCount = 0;
}

The asyncLoop() code collect the latest values from the loop() via the volatile variables and formats them and prints them to the telnet output stream.

void asyncLoop(Stream &stream) {
  if (dataTimer.justFinished()) {
    dataTimer.restart();
. . . collect volatile values here and print to telnet stream
. . .
    maxLoopTime_v = 0; // reset for next time
    stream.println();
  }
}

The asyncConnected() method is called when a new telnet connection is received. In these examples that method sends the welcome message.

void asyncConnected(Stream &stream) {
  stream.println("Results output every 2sec.");
  stream.println("millis, avg us/loop, max us/loop");
}

In ex1 the asyncDisconnected() and asyncDataReceived() methods are not used and are empty.

Connection Timeout

The asyncConnectionTimeout(unsigned long msTimeout) method sets the connection timeout. Leave as 0 (the default) if you don't regularly send data or receive commands. The method can be safely called from any core or thread, so you can call it from your Arduino setup() method.

WiFi and Ethernet connections require special handling because the connection can end up being 'half closed', which can happen went the client just disappears due to a bad WiFi connection, power loss at the router or forced shut down of the client. See Detection of Half-Open (Dropped) TCP/IP Socket Connections (or this local copy) for more details.

Step 3: A Command Example, HighSpeedESP32_ex2

This second example includes a stepper motor and three single char commands, r for Run, s for Stop and h for Home to control it. Unzip the HighSpeedESP32_ex2.zip to your Arduino sketch directory. Fill in you network SSID and password in the HighSpeedESP32_ex2.ino file. Edit the staticIP to one suitable for your network, upload to your ESP32 and connect via telnet (port 23).

In the telnet screen enter r to start the stepper running, then s for stop and h to bring it back to the start position. See the Stepper Speed Control Library for more detail on the stepper control settings.

Stepper cmds: s->stops r->runs h->sends home
Results output every 2sec.
millis,avg us/loop, max us/loop, speed,position
31618,8.59,30,0.00,0
33618,8.59,26,0.00,0
. . .
49618,9.23,32,687.69,4729
51618,9.24,36,787.41,6200
53618,9.25,31,715.56,7723
. . .
95618,9.31,30,-500.57,2006
97618,9.31,30,-400.71,1106
99618,9.30,30,-300.93,406
101618,9.12,30,0.00,0

This shows the maximum loop() time varies from ~26us to ~36us when not sending/receiving any data via Serial Monitor, On the other hand setting PRINT_DELAY_MS to 1000 at the top of HighSpeedESP32_ex2.ino and running again produces data like this.

32553,10.21,296,35.36,2313
34553,9.81,295,-12.33,2325
36553,10.27,314,-110.22,2205

Even modest logging to the Serial Monitor makes the maximum loop() time about 10 times slower.

Coding for Commands

The command handling is done in asyncDataReceived() which reads the incoming TCP data and sets the stepper control variable accordingly.

void asyncDataReceived(Stream &stream) {
  while (stream.available()) {
    char c = stream.read();
    // s for stop, r for run, h for home
    if (c == 's') {
      stepperCtrl_v = STOP;
    } else if (c == 'r') {
      stepperCtrl_v = RUN;
    } else if (c == 'h') {
      stepperCtrl_v = HOME;
    } // ignore other chars
  }
}


Step 4: A PfodApp Control and Charting Example, HighSpeedESP32_ex3

You can use the free Andriod app, pfodDesignerV3, to design a simple menu and chart for the stepper. See pfodDesignerV3 tutorials for how to use it to create menus and charts and generate the Arduino code for you.

While pfodDesignerV3 can generate native Arduino code for the ESP32, because here the WiFi communication is already coded in HS_AsyncTCP, select Serial as the Target for the pfodDesignerV3 code generation. The pfodDesignerV3 produced this code for the menu, pfodDesignerV3.txt

The updated WifiDataHandling.cpp file is contained in the HighSpeedESP32_ex3.zip. Note carefully only the millis, position, speed and max loop() us are now logged so as to match the designed chart. Here is a chart for a run which does not write to the Serial Monitor. The max loop() us is <35us

The third chart is when there is data written to the Serial Monitor once a sec. It shows the slowing of the loop() every second.

Step 5: Limitations on Data Logging Via WiFi

The data sent via WiFi is written to the async stream. The async stream has 1400 char buffer which is sent, if there is a connection, once the async... method returns. If you try to write more then 1400 chars in any one async... method calls, any extra are just discarded.

The asyncLoop() method is called about every 1.5ms by the FreeRTOS running on core 1, but if the WiFi is busy it can be 20ms or so between calls. This sets the upper limit on data logging frequency unless you buffer data in the Arduino loop() and transfer it in blocks, which is not covered in this tutorial.

The next limitation is the TCP WiFi transfer and ack times. This can vary from a few ms to tens of ms. If the ESP32 WiFi subsystem is not available to accept the next buffer of stream data it is just dropped.

See Remote High Speed Data Logging for more information and for high speed data logging to an SD card see High Frequency, Long Duration Datalogging

Conclusion

This tutorial showed how to use the HP_AsyncTCP library to completely remove the data logging and control I/O from your Arduino loop() so that is consistently runs as fast as it can. You can also use the techniques from Simple Multitasking Arduino to simulate multitasking and to give important tasks more priority in your Arduino loop() without resorting Interrupts or to the complexity and relatively slow task switching (1ms ticks) of RTOS (Real Time Operating System).