Introduction: ESP8266 Basic Server

The world of DIY electronics is very exciting! Lots of new ideas come out every day, and some of them face a common problem: how can we improve the user interface with our creations? Of course we can use our nice skills and create very complex circuits with fancy, colorful buttons or... we can use the sophisticated interface we all have at our hands: our mobile devices ;-)

There are many ways to do this, like the usual mobile apps connected to modern IoT systems, but this is not the only way. There is a single app every smartphone and tablet has: the web browser, so let's use it!

This instructable will show a simple way to create a webserver running at a ESP8266 microcontroller, intended to be used as a user interface for your creations (but, of course, there are many more uses for it).

Supplies

To run your own ESP8266 server you will need, of course, the microcontroller. This project used a Wemos D1 R1 board, a very affordable one.

Besides the board, some electronic parts were used:

  • 2 standard leds with different colors;
  • a simple button.

The source code for the server can be found at GitHub: https://github.com/lujunq/espserver

Step 1: The Circuit

The circuit used at this project is a very simple one: just 2 leds for feedback and a single button used to reset the server. Check out the image above for details (ok, that's an Arduino board, but the pins are very similar to the ones found on Wemos and other ESP8266 boards - the main difference between Arduino and Wemos pins are the analog ones: Wemos has only one).

We will use the two leds as feedback for our server state - that's why we need them in two colors. At this setup they are connected to digital pins 5 and 4.

The button will be used to reset the server configuration. It is connected to digital pin 3.

That's all for our circuit! Simple, isn't it?

Step 2: Setting Up the Code Environment

Go ahead and download the source code for our server at https://github.com/lujunq/espserver. It is intended to be used with the standard Arduino IDE (version 1.8.10 by the time this text is written). We will need, however, to add some gizmos for it to work properly with our code.

First, we need to add support for the ESP8266 microcontroller/board. This is a simple task: open the IDE preferences windows (File > Preferences). Then, at the Settings tab, look for the Additional Boards Manager URLs option. This is a list of URLs with descriptions for additional boards, besides the Arduino/Genuino ones. You can add as many URLs as needed using commas to separate them. Our code works well with this setup:

https://dl.espressif.com/dl/package_esp32_index.json, https://arduino.esp8266.com/stable/package_esp8266com_index.json

Just copy the above text to your IDE setup and click OK. You'll notice that many news boards were add at the Tools > Boards menu. For this project, the Wemos D1 R1 was selected.

The next step can be a little tricky. Since we'll be running a web server we'll need to store the files served at out board... but don't worry: no need to add storage shields or SD cards to our circuit. The ESP8266 already comes with amazing 4MB to hold our files (more than sufficient for our interface needs). We'll see how to manage files in the future - for now let's just add the tool that handles it to our IDE. Follow these steps:

  1. Download the tool. It's a JAR file that can be found here: https://github.com/esp8266/arduino-esp8266fs-plugin/releases (version 0.4.0 by the time this text is written - just download the ESP8266FS-0.4.0.zip file).
  2. Unzip the downloaded file and look for esp8266fs.jar inside it (the only file at the package).
  3. Copy the esp8266fs.jar file to your Arduino IDE installation tools folder, and this is the tricky part. Where is it? Well, if you're using the Windows version of the IDE, check your documents folder: there you'll find an Arduino directory. Create, if not exists, the folders tools/ESP8266FS/tool and put the extracted file there. At Linux and macOS systems the steps are very similar.

You'll need to restart your IDE after that. If everything went fine, you'll find the new ESP8266 Sketch Data Upload entry at the Tools menu. We'll come back to it later.

That's it. We're good to go!

Step 3: What This Server Actually Does!

Now that we have a working project loaded at the Arduino IDE, let's check out what we're trying to accomplish here. When we talk about a web server built on this kind of board, we're not only looking for the ways to deliver files/pages to web browsers. We also need to enable the devices to actually connect to it. So, we have 2 major jobs here...

Enable the connection to the user devices

The ESP8266 has a built-in wifi interface and we'll use it to communicate with our devices (phones/tablets/computers). It supports both the most common ways for this: it can start it's own access point, with it's unique SSID and password, OR it can join an existing network by connecting to your wireless router so all your gizmos can access it at once. We'll be supporting both at our project (exciting, isn't it?).

At the first boot, the system will start providing it's own network, so we'll need to connect to it with our device to access the setup interface. After that, we'll be able to change the behavior by providing the SSID and password of our router if we want to. Oh, the system is smart enough to store this info and use it next time it is started.

The leds found at the circuit are used to show the connection mode: one for the access point, the other for the existing network setup. Pressing the button for some second will clear the settings and restart the board.

Serving pages/files

OK, now the devices are connected... then what? Well, the project has a webserver configured so your phone can load them using its browser! This is where our interface will be. The code provided have some sample pages that show how to set up actions in response to the user input (like entering the router SSID and password). Then you can extend it to do whatever you want: lighten leds, start motors, annoy someone with a buzzer... your imagination is the limit!

Step 4: A Closer Look at the Code

Open the project at the Arduino IDE. As usual, the first lines include the libraries used. Then we have the project settings: a name and an id. By default this id will be used as the SSID of the access point. The next steps are the wifi setup variables.

// project settings
String prName = "ESP Server";
String prId = "espserver";

// wifi setup
bool modeAp = false;
String apSsid = prId;
String apPassword = "123456789";
String ssid = "";
String password = "";

We'll now start our webserver at the standard port 80 and set the pins used by the leds and the button.

// preparing webserver
ESP8266WebServer server(80);
const int apLed = D5;
const int staLed = D4;
const int resetBt = D3;
int busyLed;
int modeLed;

Then, our setup routine. This is a very long one compared to the usual ones from Arduino sketches, so let's break it in smaller pieces to understand it better. We'll be jumping the the sketch functions when needed to explain it better.

// initializing board
Serial.begin(9600);
Serial.println("");
Serial.println(prName + " is starting...");
pinMode(apLed, OUTPUT);
digitalWrite(apLed, LOW);
pinMode(staLed, OUTPUT);
digitalWrite(staLed, LOW);
pinMode(resetBt, INPUT);
Serial.println("");

The usual pin assignment and serial output setup (you can use the serial monitor of the Arduino IDE to check out the server state). At this time, both mode leds are turned off.

// initializing file system
SPIFFS.begin();

Here, we initialize the file system used to store the webserver files. We have 4MB - this may not sound too much, but it is enough for our needs, and we'll check out some optimizing later.

// loading saved configuration
Serial.print("Loading configuration... ");
getWifiMode();
Serial.println("");

Now, we check out our system configuration. It is stored as simple files and loaded using the following function:

// get current wifi mode settings
void getWifiMode()
{
  // do ssid and password files exist?
  if (SPIFFS.exists("/wifimode/ssid") && SPIFFS.exists("/wifimode/pass")) {
    // get info to connect to an existing wifi network
    Serial.print("configuration files found... ");
    String wfSsid = "";
    String wfPass = "";
    File fl = SPIFFS.open("/wifimode/ssid", "r");
    wfSsid = fl.readStringUntil('\r');
    fl.close();
    fl = SPIFFS.open("/wifimode/pass", "r");
    wfPass = fl.readStringUntil('\r');
    fl.close();
    // information really available?
    if ((wfSsid != "") && (wfPass != "")) {
      // connect to an wifi network
      ssid = wfSsid;
      password = wfPass;
      Serial.println("connect to wifi " + ssid + "!");
      modeAp = false;
    } else {
      // no information: start as an access point
      Serial.println("failed to load configuration files.");
      modeAp = true;
    }
  } else {
    // mode set to access point
    Serial.println("no configuration files found.");
    modeAp = true;
  }
}

Notice we store the SSID and the password of the network we'll connect into two separate files, /wifimode/ssid and /wifimode/pass. If one of these files does not exist, the system starts as an access point (notice the global modeAp variable set to true).

After we determine the connection mode, we'll jump to its initialization, back to the setup method.

// initializing wifi
if (modeAp) {
    // set led output
    busyLed = staLed;
    modeLed = apLed;
    // start as an access point
    Serial.println("Starting access point...");
    WiFi.mode(WIFI_AP);
    WiFi.softAP(apSsid, apPassword);
    IPAddress IP = WiFi.softAPIP();
    Serial.print("Access point ");
    Serial.print(apSsid);
    Serial.print(" started with password ");
    Serial.println(apPassword);
    Serial.print("Host address: ");
    Serial.println(IP);
    Serial.println("");
    digitalWrite(modeLed, HIGH);
  } else {
    // set led output
    busyLed = apLed;
    modeLed = staLed;
    // connect to an existing wifi network
    Serial.print("Starting wifi");
    WiFi.mode(WIFI_STA);
    WiFi.begin(ssid, password);
    while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
    }
    Serial.println("");
    Serial.print("Connected to ");
    Serial.println(ssid);
    Serial.print("IP address: ");
    Serial.println(WiFi.localIP());
    Serial.println("");
    digitalWrite(modeLed, HIGH);
  }

At this point, we are checking the modeAP variable to determine how to proceed. If we have an access point, we'll use the WiFi object static method to start broadcasting our network (check out the serial output while running the software). Notice the mode leds are also initialized properly. If we are connecting to a router, we will wait until the connection is completed to proceed. In that case, the serial monitor will show us the IP assigned to our board.

Next, we'll start the mDNS protocol (multicast DNS) to simplify the connection to our board. If you're not familiar with it, mDNS enables you to access a single device on your network without the need to know its assigned IP. I know, this is magical! Of course there's much more about mDNS: if you're interested, check out this post at Wikipedia.

// initializing mDNS
Serial.println("Starting mDNS");
if (MDNS.begin(prId)) {
  Serial.println("mDNS started at http://" + prId + ".local - no Android support, sorry :-(");
} else {
  Serial.println("failed to start mDNS - sorry :-(");
}
Serial.println("");

With the code above, we're creating the http://espserver.local address at our network so we can easily access the server. More about that "no Android support" latter...

The next step is to assign the actions for our webserver. This is how it works: if you assign a custom action to a page/request (let's call it a "route"), when it is accessed by the browser, a custom function of your sketch is called. If a non-assigned route is called, the server will just look for the appropriate file and send it back to the user. Let's take a look at the first custom assignment:

// example custom routing action
server.on("/example.html", onExample);

The code above tells the server to run the function onExample every time a browser requests the file example.html. Here is the function:

// example routing action
void onExample()
{
  // preparing variables
  String bgColor = "white";
  // looking for expected arguments
  if (server.arg("color") == "red") bgColor = "red";
  if (server.arg("color") == "blue") bgColor = "blue";
  // showing appropriate HTML file
  sendFile("/example" + bgColor + ".html");
}

This is a simple function, but it shows a lot. So you accessed example.html at your phone... what will happen? Well, it will start by checking out the background color you defined. This is done by checking the received GET variables of your HTTP request. If you're not familiar with this, Wikipedia is your friend ;-) To simplify, GET variables are sent with the link you are accessing with your web browser after a ? sign, like example.html?color=red or example.html?color=blue. We use the server.arg function to check the value of a variable received at the request. At our example, we just look for a HTML file with the appropriate background color and return it to the user. Notice that we don't exactly return a file named example.html to the browser, we select another one and send as if it was the one it requested. Of course you can create very complex behaviors for your custom actions. This is just a simple example one.

Following the setup routine, we define the action for the index.html route. Notice that we assign the same action when the user access the server without providing any route (the / sign assignment).

// basic index route
server.on("/", onIndex);
server.on("/index.html", onIndex);

The last custom assignment is for the wifi setup. It uses a similar approach we did for the example route to set the connection mode and router SSID/password. Here is the custom action function:

// wifi setup
void onWifi()
{
  // action set?
  if (server.arg("ac") == "ap") {
    // change wifi mode to access point
    setWifiModeAp();
  } else if (server.arg("ac") == "sta") {
    // connection to an existing wifi network
    String wfSsid = server.arg("ssid");
    String wfPass = server.arg("password");
    if ((wfSsid != "") && (wfPass != "")) {
      setWifiModeSta(wfSsid, wfPass);
    }
  }
  // show wifi interface
  sendFile("/wifi.html");
}

// set the connection mode to an access point
void setWifiModeAp()
{
  if (SPIFFS.exists("/wifimode/ssid")) SPIFFS.remove("/wifimode/ssid");
  if (SPIFFS.exists("/wifimode/pass")) SPIFFS.remove("/wifimode/pass");
  Serial.println("Wifi mode changed to access point.");
  resetFunc();
}

// set the connection mode to wifi network client
void setWifiModeSta(String wfSsid, String wfPass)
{
  File fl = SPIFFS.open("/wifimode/ssid", "w+");
  if (!fl) {
    // can't set wifi mode: nothing to do...
    Serial.println("Can't change wifi settings. Using access point.");
  } else {
    fl.println(wfSsid);
    fl.close();
    File fl = SPIFFS.open("/wifimode/pass", "w+");
    if (!fl) {
      // can't set wifi mode: nothing to do...
      Serial.println("Can't change wifi settings. Using access point.");
    } else {
      fl.println(wfPass);
      fl.close();
      Serial.println("Wifi mode changed: connect to " + wfSsid + ".");
    }
  }
  resetFunc();
}

Notice, at the code above, how we write/delete files from the built-in storage provided by the ESP8266. Also, check out the resetFunc we use to reset the board after the setup:

// reset board function
void(* resetFunc) (void) = 0; //declare reset function @ address 0

Finally, we assign the server last action, and this is a special one: this is the function that will be called when the requested route has no custom setup:

// no assigned route found: try to send the corresponding file to client
server.onNotFound(onNoRoute);

As told before, when this happens we just look for the file and send it back to the browser. Oh, and server.uri() is the function you can use to check out what route the browser requested.

// no defined route: just send the corresponding file
void onNoRoute()
{
  // just try to send the requested file
  sendFile(server.uri());
}

Did you notice that sendFile function? It appeared before at the other custom actions. It is used to select a file at the built-in storage and send it back to the browser. Check it out:

// sending a file to the connected client
void sendFile(String path)
{
  // only lower case
  path.toLowerCase();
  // visual output
  digitalWrite(busyLed, HIGH);
  // does the route file exist?
  if(SPIFFS.exists(path) || SPIFFS.exists(path + ".gz")) {
    // check mime type
    String mime = "text/html";
    bool textfile = true;
    if (path.endsWith(".css")) mime = "text/css";
    if (path.endsWith(".js")) mime = "application/javascript";
    if (path.endsWith(".txt")) mime = "text/plain";
    if (path.endsWith(".jpg")) {
      mime = "image/jpeg";
      textfile = false;
    }
    if (path.endsWith(".jpeg")) {
      mime = "image/jpeg";
      textfile = false;
    }
    if (path.endsWith(".png")) {
      mime = "image/png";
      textfile = false;
    }
    if (path.endsWith(".gif")) {
      mime = "image/gif";
      textfile = false;
    }
    // sending file to client
    if (textfile) {
      if (SPIFFS.exists(path + ".gz")) {
        // sending compressed text file
        File txt = SPIFFS.open((path + ".gz"),"r");
        server.streamFile(txt, mime);
        txt.close();
      } else {
        // sending plain text file
        File txt = SPIFFS.open(path,"r");
        server.streamFile(txt, mime);
        txt.close();
      }
    } else {
      // send binary file
      File fl = SPIFFS.open(server.uri(), "r");
      server.streamFile(fl, mime);
      fl.close();
    }
  } else {
    // file not found
    String message = "ROUTE ERROR\n\n";
    message += "route: ";
    message += path;
    message += "\nmethod: ";
    message += (server.method() == HTTP_GET) ? "GET" : "POST";
    message += "\narguments: ";
    message += server.args();
    message += "\n";
    for (uint8_t i = 0; i < server.args(); i++) {
      message += " " + server.argName(i) + ": " + server.arg(i) + "\n";
    }
    server.send(404, "text/plain", message);
  }
  // visual output
  digitalWrite(busyLed, LOW);
}

The function starts by converting the route/path to lower case (to simplify our work). Then we give the user a visual feedback by turning on one of the leds (the one not used to show connection mode) - we will turn it off when the file processing is finished, giving the user a nice blinking led feedback for server access.

Then, we will check if the file exists at our storage. We'll be looking for a file with the exact path/name we received or one ending with ".gz" (more about this later). If the file is found, we'll need to determine its MIME type (don't know what MIME type is? again, check out Wikipedia). After that, we just need to open, read, send and close the file. If the file is not found, we just send an error message back to the browser. When finished, the led is turned off...

After the setup we have the usual loop method. This one is simple:

// sketch running
void loop() { // check out server requests server.handleClient(); MDNS.update(); // checking the reset button if (digitalRead(resetBt) == HIGH) { // wait for 1 second and check again delay(1000); // still pressed? if (digitalRead(resetBt) == HIGH) { // wait for another 1 second delay(1000); // still pressed? if (digitalRead(resetBt) == HIGH) { // user really wants to reset configuration Serial.println("RESETING CONFIGURATION!"); digitalWrite(modeLed, LOW); setWifiModeAp(); } } } }

The first thing done here is to handle client connections to our server. Two function calls are enough server.handleClient and MDNS.update. Then, we just need to check the reset button pressing and we're done.

Well, that's it. A little long for an Arduino sketch, but remember: to use it at your own projects, you'll just need to add our custom route processing actions (and remove the example one).

Step 5: Issues (sort Of)

We have two options to connect our devices to the server: using it as access point or from a wireless router. On both cases there are some issues you better know.

Access point

When the access point is used, you just need to set the connection on your device (SSID/password) and access the default board IP at its browser: http://192.168.4.1/

However, if you're using an Android device (phone/tablet/etc) you may end up receiving an error while doing it. This happens because the way the ubiquitous system handles wireless connections. If the wifi network it is connected doesn't have Internet access (the case of our board) and it has another connection, like a mobile 4G network, it will just ignore the wifi and won't access local addresses. This is sad, I know... There is a workaround, however: temporarily disable web connection from the mobile network and try again (this may be as easy as dragging down the upper tray of your Android and touching the 4G connection to turn if off). Devices running iOS/iPadOS, like iPhones and iPads, don't have this problem.

Using a router

When you set your board to connect to a router instead of creating an access point, the mDNS (multicast DNS) helps you to access your board without the need to know its assigned IP. It is simple as typing http://espserver.local at your browser.

However... here comes the Android again... The Google system does not support multicast DNS, so you'll need to type the IP. Sorry. Other systems like iOS, Windows, Linux or macOS work pretty well with mDNS.

Step 6: Handling Files

The ESP8266 comes with 4MB for your files. This may sound too little, but it is enough for our needs.

This storage is handled using the SPI Flash File System (or just SPIFFS). We have previously downloaded and installed a tool to handle file uploading to our board, so let's learn how to use it! First of all, check out the Arduino IDE project you downloaded from GitHub. You'll find two folders on it: data and html. Let's discuss the data one for now (html in just a moment).

The ESP8266 Sketch Data Upload tool

Look for the ESP8266 Sketch Data Upload entry of your Arduino IDE Tools menu. Every time you access it, the built-in storage of your ESP8266 board is cleared and replaced by the content of the data folder (provided that the board is connected to your IDE, of course). It is as simple as that!

Since the folder structure is kept during this upload, you can use the data folder to set your server routes. Save a file named index.html at the root of the data folder and it will be available at the /index.html route of your server. Save another one named picture.png inside a img folder of your data, and it will be available to the server at the /img/picture.png route. Simple, isn't it?

However, this comes with a drawback: since we store our configurations using the built-in storage, every time we upload files to the board we'll loose them.

Optimizing the ESP8266 server

What this tiny microcontroller does is, indeed, impressive! It provides us with the way to connect our devices and to serve web pages. However we must remember its limitations. This is a resource-limited hardware and, of course, won't work as powerful systems, like Linux machine running an Apache server. Because of that we must consider some optimizations.

First, while running as an access point, remember it will not handle a lot of simultaneous connected devices like fully-flagged routers. Four clients would be the limit, but consider not using more than two.

About the webserver, it may fail serving a lot of large HTML/CSS/JS files along the associated images. A good way to make things easier for your microcontroller is compressing the text files and sending them gzipped to the browser. Yes, all browsers handle this. Remember the sendFile function of our code? It does look for ".gz" versions of the files to send them back to the user. Look at the html folder of the project: it comes with the uncompressed versions of the files, while you'll find the gzipped ones at the data folder, the ones uploaded to the board. You may use any software that can handle gzip compress to do this. 7Zip is a good choice. Remember: GZIP, not the usual ZIP format.