Load Your Arduino/ESP Config Webpage From the Cloud

When creating an Arduino / ESP (ESP8266/ESP32) project, you could just hardcode everything. But more often than not something turns up and you'll end up re-attaching your IoT-device to your IDE again. Or you just got more people accessing the configuration and you want to provide an UI instead of expecting them to understand the inner workings.

This instructable will tell you how to put most of the UI in the cloud instead of on the Arduino / ESP. Doing it this way will save you on space and memory usage.
A service providing free static webpages is particularly suitable as "cloud", like GitHub Pages, but other options will probably work as well.

Building up the webpage this way requires the user's browser to go through 4 steps:

1. Request the root url from the Arduino / ESP
2. Receive a very simple webpage, telling to:
3. Request a JavaScript file from the cloud
4. Receive the code that builds up the actual page

This Instructable will also explain how to interact with the Arduino / ESP once the page is ready as per the above steps.

The code created on this instructable can be found on GitHub as well.

Prerequisites

This instructable assumes you got access to certain materials and some prior knowledge:

• An Arduino (with network acces) / ESP
• A computer to attach the above to
• WiFi access connected to the internet
• The Arduino IDE installed (also for the ESP32)
• You know how to upload a sketch to your IoT-device
• You know how to use Git & GitHub

Step 1: Starting Out With a Simple Webserver Sketch

We'll start as simple as possible, and let it grow from here on.

#include <WiFi.h>

const char* ssid     = "yourssid";
const char* password = "yourpasswd";

WiFiServer server(80);

void setup()
{
  //Initialize serial and wait for port to open:
  Serial.begin(115200);
  while(!Serial)
  {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  WiFi.begin(ssid, password);

  while(WiFi.status() != WL_CONNECTED)
  {
    delay(500);
    Serial.print(".");
  }

  Serial.println("WiFi connected.");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
  
  server.begin();
}

void loop()
{
  // listen for incoming clients
  WiFiClient client = server.available();   // listen for incoming clients
  bool sendResponse = false;                // set to true if we want to send a response
  String urlLine = "";                      // make a String to hold the requested URL

  if(client)                                // if you get a client,
  {
    Serial.println("New Client.");          // print a message out the serial port
    String currentLine = "";                // make a String to hold incoming data from the client
    while(client.connected())               // loop while the client's connected
    {
      if(client.available())                // if there's bytes to read from the client,
      {
        char c = client.read();             // read a byte, then
        
        if(c == '\n')                       // if the byte is a newline character
        {
          // if the current line is blank, you got two newline characters in a row.
          // that's the end of the client HTTP request, so send a response:
          if(currentLine.length() == 0)
          {
            sendResponse = true;             // everything's fine!
            break;                           // break out of the while loop
          }
          else                               // if you got a newline, then clear currentLine:
          {
            if(currentLine.indexOf("GET /") >= 0) // this should be the URL line
            {
              urlLine = currentLine;         // save it for later use
            }
            currentLine = "";                // reset the currentLine String
          }
        }
        else if(c != '\r')                   // if you got anything else but a carriage return character,
        {
          currentLine += c;                  // add it to the end of the currentLine
        }
      }
    }
    
    if(sendResponse)
    {
      Serial.print("Client requested ");
      Serial.println(urlLine);
      
      // HTTP headers always start with a response code (e.g. HTTP/1.1 200 OK)
      // and a content-type so the client knows what's coming, then a blank line:
      client.println("HTTP/1.1 200 OK");
      client.println("Content-type:text/html");
      client.println();
      
      if(urlLine.indexOf("GET / ") >= 0) // if the URL is only a "/"
      {
        // the content of the HTTP response follows the header:
        client.println("Hello world!");
      }
      
      // The HTTP response ends with another blank line:
      client.println();
    }
    
    // close the connection:
    client.stop();
    Serial.println("Client Disconnected.");
  }
}

Copy over the above code, or download it from the commit on GitHub.

Don't forget to change the SSID and password to match your network.

This sketch uses the well known Arduino setup() and loop() functions.
In the setup() function the serial connection to IDE is initialized and so is the WiFi. Once the WiFi is connected to the said SSID the IP is printed and the webserver started.
In each iteration of the loop() function the webserver is checked for connected clients. If a client is connected the request is read and the requested URL is saved in a variable. If everything seems alright, a response from the server to the client is performed based on the requested URL.

WARNING! This code uses the Arduino String class to keep it simple. String optimizations are not within the scope of this instructable. Read more about this on the instructable about Arduino String Manipulation Using Minimal Ram.

Step 2: Creating the Remote JavaScript

The Arduino / ESP will tell the visitors browser to load this one file. All the rest will be done by this JavaScript code.

In this Instructable we'll make use of jQuery, this is not strictly necessary, but will make things easier.

This file needs to be accessible from the web, a static pages server is enough to make this work, for example GitHub pages. So you'll probably want to make a new GitHub repository and create a gh-pages branch. Put the following code inside a .js file in the repository in the correct branch.

var cdnjsURL = 'https://cdnjs.cloudflare.com/ajax/libs/',
    $;
(function()
{
    var script = document.createElement('script'); // create a <script> element
    script.src = cdnjsURL + 'jquery/3.2.1/jquery.min.js'; // set the src="" attribute
    script.onload = function() //callback function, called once jquery file is loaded
    {
        $ = window.jQuery; // make jQuery accessible as the global $ variable
        init(); // call the init function
    };
    document.getElementsByTagName('head')[0].appendChild(script); // add the created <script> tag to the document, this will start loading of the jQuery lib
})();

function init()
{
    // Done loading jQuery, will add code here later on...
}

Copy over the above code, or download it from the commit on GitHub.

Check if your file is accessible. In case of GitHub pages go to https://username.github.io/repository/your-file.j... (replace username, repository and your-file.js for the correct parameters).

Step 3: Loading the Remote JavaScript File Into the Visitors Browser

Now that we've got everything set up it's time to make the webpage load the remote JavaScript file.

You can do this by changing line 88 of the sketch from

client.println("Hello world!");

to

client.println("<!doctype html><html><head><script src=\"https://username.github.io/repository/your-file.js\"></script></head><body></body></html>");

(change the src attribute to point to your own JavaScript file).
This is a small html webpage, all it does is load the JavaScript file into the visitors browser.

The altered file can also be found in the corresponding commit on GitHub.

Upload the adjusted sketch to your Arduino / ESP.

Step 4: Adding New Elements to the Page

An empty page is useless, so it is now time to add a new element to the webpage. For now this will be a YouTube video.
In this example some jQuery codes will be used to accomplish this.

Add the following line of code to the init() function:

$('<iframe>').prop({ src: 'https://www.youtube.com/embed/k12h_FOInZg?rel=0', frameborder: '0' }).css({ width: '608px', height: '342px' }).appendTo('body');

This will create an iframe element, set the correct src attribute and set the size using css and add the element to the body of the page.

jQuery helps us to easily select and alter elements in the webpage, some basic things to know:

• $('body') selects any already existing <body> element, other css selectors can be used as well
• $('<div>') creates a new <div> element (but does not add it to the document)
• .appendTo('.main') appends the selected/created element to an element with css class 'main'
• Other function to add elements are .append(), .prepend(), .prependTo(), .insert(), .insertAfter(), .insertBefore(), .after(), .before()

Take a look at the corresponding commit on GitHub if anything is unclear.

Step 5: Interactive Elements

A video is fun, but the purpose of this instructable is to interact with the Arduino / ESP.
Let's replace the video for a button that sends information to the Arduino / ESP and also waits for a response.

We'll need a $('<button>') to add to the page, and attach an eventlistener to it. The eventlistener will call the callback function when the specified event happens:

$('<button>').text('a button').on('click', function()
{
  // code here will be executed when the button gets clicked
}).appendTo('body');

And add an AJAX request to the callback function:

$.get('/ajax', function(data)
{
  // code here will be executed when the AJAX request is finished
});

Once the request is finished, the returned data will be added to the page:

$('<p>').text(data).appendTo('body');

In summary, the above code creates a button, adds it to the webpage, when the button is clicked a request will be send out and the response will also be added to the webpage.

If this is your first time using callbacks you might want to check the commit on GitHub to see how everything is nested.

Step 6: Respond to the Interactive Element

Of course, the AJAX request requires a response.

To create the correct response for the /ajax url we'll need to add an else if() right after the closing bracket of the if statement that checks on the / url.

else if(urlLine.indexOf("GET /ajax ") >= 0)
{
  client.print("Hi!");
}

In the commit on GitHub I've also added a counter to show the browser that every request is unique.

Step 7: Conclusion

This is the end of this instructable. You now have an Arduino / ESP serving a small webpage that tells the visitor's browser to load a JavaScript file from the cloud. Once the JavaScript is loaded it builds up the rest of the content of the webpage providing a UI for the user to communicate with the Arduino / ESP.

Now it is up to your imagination to create more elements on the webpage and save the settings locally on some kind of ROM (EEPROM / NVS / etc).

Thank you for reading, and please feel free to give some feedback!