Introduction: Fetch: the Easiest Way to Make HTTP Requests From Your Arduino and ESP8266/ESP32.

By Check out Grandeur!
About: Simplifying IoT dev; abridging that huge gap between our softwares & hardwares! | Habits over rules. Exploration.

I got a request from a follower redditer the other day, asking for help with an HTTPS (the 'S' is important here!) POST request code that he wrote. He wrote it by editing a GET request code that he found on the internet, but his code won't work. Those who are familiar with the web know that a POST request is very different from the basic GET request, because now you have a body and there are certain ways to write the body (URL encoded or JSON or Multipart Form Data) according to your application, and whichever way you are using, you need to specify that in the request headers using the Content-Type header and the length of the body using the Content-Length header (this one is optional though!). And you need to space the body from the headers by an additional "\r\n", and then another "\r\n\r\n" after the body to show the end of the request. Exhausting, believe me I know!

Here's the code that I sent him for his ESP8266:

#include "recipes/WiFi.h"#include <WiFiClientSecure.h>#define SSID YOUR_WIFI_SSID#define PASSPHRASE YOUR_WIFI_PASSPHRASE// Whatever server you want to send the request to.#define URL "api.grandeur.tech"#define PORT 443 // HTTPS port#define PATH "/auth/login/?apiKey=grandeurkywxmoy914080rxf9dh05n7e"#define METHOD "POST"#define CONTENTTYPE "application/json"#define BODY "{\"email\": \"test@test.com\", \"password\": \"test:80\"}"const char fingerprint[] PROGMEM = "DC 78 3C 09 3A 78 E3 A0 BA A9 C5 4F 7A A0 87 6F 89 01 71 4C";void setup() {    Serial.begin(9200);    // Connect to WiFi.    connectWiFi(SSID, PASSPHRASE);}void loop() {    WiFiClientSecure httpsClient;    // Setting the fingerprint for SSL.    httpsClient.setFingerprint(fingerprint);    httpsClient.setTimeout(15000);    delay(1000);    Serial.print("Connecting to ");    Serial.println(URL);    // Forming a secure connection with the server before making the request.    while(!httpsClient.connect(URL, PORT)) {        delay(1000);        Serial.print(".");    }    // Forming the request (the hardest part).    String request =        String(METHOD) + " " + PATH + " HTTP/1.1\r\n" +        "Host: " + URL + "\r\n" +        "Content-Type: " + CONTENTTYPE + "\r\n" +        "Connection: close\r\n\r\n" +        BODY + "\r\n\r\n";    // Printing the request to be sure it's formed fine.    Serial.println("Request is: ");    Serial.println(request);    // Making the request.    httpsClient.print(request);    // Receiving response headers.    while(httpsClient.connected()) {        String line = httpsClient.readStringUntil('\n');        Serial.println("-----HEADERS START-----");        Serial.println(line);        if(line == "\r") {        Serial.println("-----HEADERS END-----");            break;        }    }    // Receiving response body.    while(httpsClient.available()) {        String line = httpsClient.readStringUntil('\n');        Serial.println(line);    }    delay(2000);}

He was not setting up the Content-Type header while forming his request because he edited the code for a GET request where that isn't required.

This code is based directly on the WiFiCientSecure library because I found it to be the only way to make a secure HTTPS request on the whole Arduino Platform. HttpClient and arduinoHttpClient are the other two general purpose request making libraries that I found but they don't support HTTPS. This is the 2020s, I know we have memory constraints on the microcontrollers but not supporting HTTPS is unrealistic. Every server on the Web now must communicate on a secure channel using HTTPS, yet there is no client library that simplifies the requesting and also supports HTTPS.

Supplies

  1. Arduino, ESP8266, and ESP32
  2. A simple laptop

Step 1: Fetch: an Inspiration From the Browser

When making requests from the browser, any web developer would agree that the most simple yet powerful way is to use fetch:

const response = await fetch("https://api.grandeur.tech/auth/login/", {    method: "POST",    body: '{"email": EMAIL, "password": PASSWORD}'});console.log(response);

It automatically sets the appropriate headers according to URL and the data being sent. On Arduino, the first line of the above snippet is equivalent to writing:

String request = String("POST") + " " + "/auth/login/" + " HTTP/1.1\r\n" +"Host: " + "api.grandeur.tech" + "\r\n" +"Content-Type: " + "application/json" + "\r\n" +"Connection: close\r\n\r\n" +"{\"email\": \"EMAIL\", \"password\": \"PASSWORD\"}" + "\r\n\r\n";httpsClient.print(request);

And the httpsClient on Arduino doesn't return you the response as elegantly as fetch, you have to manually read the headers line by the line, then break the loop when a double newline occurs, and then read the body from thereon.

Here's what the second line of the top snippet is equivalent to on Arduino:

// Receiving response headers.while(httpsClient.connected()) {    String line = httpsClient.readStringUntil('\n');    Serial.println(line);    if(line == "\r") {        break;    }}// Receiving response body.while(httpsClient.available()) {    String line = httpsClient.readStringUntil('\n');    Serial.println(line);}

That's when I knew we need something as simple as fetch in the Arduino world as well.

Step 2: Fetch for Arduino

Here's how I imagined it on Arduino:

Response response = fetch(const char* URL, RequestOptions options);Serial.println(response);

RequestOptions should have default values so you can make a GET request by doing something as simple as this:

Response response = fetch(const char* URL);

And it should set the appropriate headers by looking at the URL (the Host and Origin header) and by looking at the body being sent (the Content-Type and Content-Length header).

But it should let the developer do powerful stuff like setting a cookie for Auth sessions and keeping the connection alive (by setting the Connection header) for Server-Sent Events.

And the response should right away tell you the status of the request (200 OK or 400 User Error or 500 Server Error) and give you the body in one function call rather than a while loop that's reading it from the socket line by line.

And it MUST support HTTPS!!!

So I made a bare-minimum library that could do all of this under the constraints of the microcontroller environment. Here's how it makes the above POST request:

#include "recipes/WiFi.h"#include "Fetch.h" #define SSID YourWiFiSSID#define PASSPHRASE YourWiFiPassphrase#define FINGERPRINT "DC 78 3C 09 3A 78 E3 A0 BA A9 C5 4F 7A A0 87 6F 89 01 71 4C"void setup() {    Serial.begin(9200);    connectWiFi(SSID, PASSPHRASE);    // Setting request options.    RequestOptions options;    options.method = "POST";    options.headers["Content-Type"] = "application/json";    options.body = "{\"email\": \"test@test.com\", \"password\": \"test:80\"}";    options.fingerprint = FINGERPRINT;    // Making the request.    Response response = fetch(        "https://api.grandeur.tech/auth/login/?apiKey=grandeurkywxmoy914080rxf9dh05n7e",        options    );        Serial.println(response);}void loop() {}

You can set the options.method to any of the GET, POST, HEAD, PUT, DELETE.

The default Content-Type header value is application/x-www-form-urlencoded but you get the flexibility to manually set any headers with options.headers object.

You can pass the body to the request by setting the options.body string.

And for an HTTPS connection, you can also pass in the fingerprint to the options.fingerprint string.

This bare-minimum effort does not yet have the automatic Content-Type guessing feature, but I'll plug it in as I get some free time from work! That would knock out another mistake that a developer like me can make while writing RequestOptions.

The Response of the request prints like a standard JSON response:

{    "ok": 1,    "status": 200,    "statusText": "OK",    "body": "{\"code\": "LOGGED-IN"}"}

And you can also access the response headers:

response.headers["Content-Type"];response.headers["Content-Length"];response.headers["Connection"];


Update

As of July 18, 2022, V0.1.2 is released, which now supports making asynchronous requests. You can pass a callback function to fetch, and instead of blocking the code until it receives the response, fetch returns instantly after making the request. When it receives the response from the internet, it passes the responses to your callback function.

void handleResponse(Response response) {    Serial.println(response);}// Making the request.FetchClient client = fetch(    "https://api.grandeur.tech/auth/login/?apiKey=grandeurkywxmoy914080rxf9dh05n7e",    options,    handleResponse);

Step 3: Conclusion

I wrote Fetch for Arduinoso I can make requests from my ESP by avoiding as many mistakes as possible. The bigger the surface area of the code, the greater the possibility that I'd miss a header or mistype a character somewhere and it'd become a debugging nightmare for me.

I hope it eases your life as much as it eases mine. I'd be importing more features from the browser's fetch as time goes on to support as many use cases as possible. Would really appreciate your feedback and support!