Introduction: ESP8266 WiFi OTA Remote File Management

By

I have been working with ESP8266 micro controllers long enough now to have a number of them deployed in my environment. I have a couple setup testing my MQTT broker(s). I have a couple sending temperature, humidity and barometric pressure readings to the cloud. I have some running for debugging and testing on my bench. I even have one running my irrigation system.

In order to make any changes to the software on any of these controllers, I have to physically attach a FTDI and USB cable to a computer and connect via COM port. This can become very problematic for ESPs deployed in hard to get to or just inconvenient locations. Its also time consuming.

In my minds eye, I can see the possibility of deploying dozens or even hundreds of these inexpensive devices to control/monitor a myriad of situations. Making changes to the software of many of these devices by physically connecting to each is impractical and will keep these great little controllers from reaching their full potential in the world of IoT.

However, with the proper tools, it is relatively easy to manage these controllers over their Internet/LAN connection!

With NodeMCU, it is possible to create and deploy a remote file management system. With the ability to remotely upload, remove, compile and run files, many more options become available. Such as, the ability to push new or updated software to these controllers in mass, remotely or OTA (over the air).

This Instructable will show you how to manage your ESP8266s over WiFi.

Step 1: Overview

Briefly, the work flow is as follows:

PHP files served via web browser will collect the users intentions to either: Read the controller status, upload a file, reboot the controller, remove files, compile files or run files on the ESP.

Once the command is collected from the user, PHP formats the data/information and sends it to the ESP via fsocket.

When the data/information is received at the ESP, LUA files parse the data and carry out the commands.

The possibility of managing ESPs over WiFi rests with the fact that the dofile() command loads the script into memory. Once in memory, the file can be removed or replaced in the ESPs file system with an edited file.

As an example. the init.lua file is automatically loaded into memory on boot up. If there is a command in this file to blink a LED or read a pin, the ESP will continue to carry out those tasks even if the init.lua file is deleted from the ESP. If a new init.lua file is uploaded or sent via WiFi that doesn't contain commands to blink a LED, nothing will happen until the file is executed ( dofile() ) or the ESP is rebooted. Then the new init.lua file will be executed and the LED will stop blinking.

We will use a web browser to send the new init.lua file and a command to restart.

There are some caveats to this solution, we will cover those later.

What you will need:

  • A webserver running on the same LAN as your ESP8266 or the ESP8266 exposed to the Internet.
  • Access to the webserver's root. You will need to upload the PHP files to a root directory in under www/html (apache) or \Inetpub\wwwroot (iis). Simple website business.
  • An ESP8266 flashed with NodeMCU and the ability to upload initial files via an IDE.
  • Some PHP and LUA understanding.
  • Source files for this project.

Step 2: Communication With the ESP8266

The ESP running NodeMCU has a number of ways to communicate with the outside world. TCP/UDP Client, TCP/UDP Server, telnet and MQTT. Since we will be sending simple text strings to the ESP, any of these options can be used.

MQTT would be a great solution for this project. However MQTT has a few more requirements, we will be using NodeMCUs TCP Server.

An ESP flashed with NodeMCU and configured as a TCP Server is capable of sending and receiving data to and from a web browser. This allows us to send files and commands to the ESP and also to query the ESP for certain information. For our purposes, when queried, the ESP can tell us how much memory is left and list the files in its file system.

Sending data to the ESP TCP server can be sent via number of methods. Some of the more conventional are 'GET' and 'POST'. Both of these include lots of header information and other information data of no use to us. The GET protocol is limited 256 characters in most configurations. ~256 characters would be a small .lua file. Although it could be done, sending files via multiple GETs would be difficult and error prone. These methods are slow and browsers can be troublesome.

PHP's fsockopen command is the tool of choice for this project. It is a one to one connection and very fast. We write to the socket just like writing to a file. And!! it only sends data written to it. No header info!

There is a limit to the number of characters sent in a fsockopen session. This is controlled by different TCP layer and most connections are limited to ~1460 characters. If the lua file we are sending is larger that 1200 characters (ASCII), our PHP file will close the session, open another, and continue to send the next 1200 characters of the file until EOF is reached. The LUA file on the ESP will either write or append to the file as it receives the data.

Step 3: The PHP Side

The PHP part of the project resides on the web server attached to the common LAN. This collection of scripts format the data and manage the transfer to the ESP. Files in /filbin folder can be sent to the ESP.

The php files for this project consist of:

  • index.php -- The GUI file for the user to send files/commands to the ESP.
  • controllerIP.txt -- Text file containing the IP number of the target ESP.
  • writeIP.php -- Saves the IP number to controllerIP.txt
  • send_Compile.php -- Sends the filename and 'compile' command to ESP
  • send_Delfile.php -- Sends the filename and 'delete file ' command to ESP
  • send_Dofile.php -- Sends the filename and 'dofile' command to ESP
  • send_file.php -- Sends the filename, command to write or append and a string containing the file to ESP
  • send_Restart.php -- Sends a restart command to ESP

These files along with an empty sub-directory called 'filebin' will need to be uploaded to the server root directory of the machine hosting your web server.

The directory /filebin contains files to be uploaded the ESP are stored.

In my previous example discussing the init.lua file blinking a LED, the /filebin directory is where you will put the edited/new version of the init.lua file. When you are finished editing, simply click on the file name and it will be sent to the ESP.

The index.php file (image above) uses simple forms and buttons to send information to the associated php execute file.

Clicking the "ReBoot ESP" button sends the IP number to the send_Restart.php file. The send_Restart.php file opens the socket to the IP, formats the message "**command**restart**" and writes that string to the ESP. META http-equiv= refresh is used to redirect back to index.php after 5 seconds.

    All the commands are similar with the exception of writeIP and Controller Status.

    The 'Controller Status' button is simply linked to the controllers IP number with "/status" appended to the IP.

    Clicking on any of the files in "Files in '/filebin' available for upload:" area sends the filename and IP number to the 'send_file.php' file.

    The 'send_file.php' reads the target file into an array, then begins to reassemble the file into one string prepended with the command "**command**Newfile**\n". If/when the string exceeds 1200 characters the socket is closed. Another socket is opened and the string continues, this time pre-pended with "**command**Apdfile**\n"

    Step 4: The ESP8266 LUA Side

    The .lua files on the ESP8266 do the heavy lifting of parsing the strings coming over via fsockets. They 'catch' the payload (pl) coming from the webserver and parse for instructions.

    In this example four files are required:

    • init.lua
    • servernode.lua
    • wifi_tools.lua
    • getstatus.lua

    Initially, these files have to be uploaded to the ESP via FTDI and COM port.

    init.lua, servernode.lua and wifi_tools.lua must ALWAYS be in the file system of the ESP. They have to be available to "dofile()".

    This is one of the caveats mentioned earlier.

    When the ESP boots, it MUST to boot into a server mode and it MUST be able to access "dofile()" wifi_tools.lua.

    If you 'format' the controller, its back to a FTDI and a COM port. Yes, you can format via WiFi.

    With the exception of "wifi_tools.lua" and "servernode.lua", these files can be removed and uploaded between reboots.

    !! "wifi_tools.lua" and "servernode.lua" can only be overwritten, never removed !!

    The size of "wifi_tools.lua" and "servernode.lua" normally, cannot exceed 1200 bytes (ASCII characters, including whitespace and linefeeds)

    If "wifi_tools.lua" is larger that 1200 bytes, it will be partially written and will cause 'panic' when the ESP tries to dofile from the servernode.lua to append the rest of the file.

    "servernode.lua" is a generic TCP server file, code can be found on the NodeMCU api site with this addition:

    • if string.sub(pl, 0, 11) == "**command**" then
    • dofile("wifi_tools.lua")
    • end
    • if string.find(pl, "status") then
    • dofile("getstatus.lua")
    • tmr.delay(250)
    • file.open("info.lua", "r")
    • conn:send(file.read())
    • file.close("info.lua")
    • end

    The variable "pl" is the payload from the fsocket connection, this is the string generated by the PHP files server-side.

    If **command** is found than the data string is parsed by wifi_tools.lua for exact details and commands.

    The 'getstatus.lua' routine is a bit convoluted, however it allows us to see which files are actually on the ESP in a near real time fashion. It also demonstrates the ability to create and serve files dynamically.

    Step 5: Final Thoughts and Source File Information.

    There is a learning curve associated with this project and stable managing of the ESP. One has to bear in mind how files are affected by other files.

    For example, in this setup the init.lua calls the servernode.lua file, the servernode.lua file may call the wifi_tools.lua file. If this chain is broken, the ESP crashes and restarts.

    If init.lua or servernode.lua are causing the panic, the ESP will loop forever. Probably nothing short of re-flashing the chip will cure this.

    Only add the dofile() command when you are sure its working correctly by manual debugging.

    It is good practice to run files manually during debugging to avoid these loops. With this project that isn't always possible and some other form of debugging must be employed. Many print() commands come in handy. Adding print commands and other debugging code to wifi_tools.lua is not a problem. The file can be as large as you like.

    In fact you can add more commands to such as starting and stopping timers and even Format if you like. :)

    Just remember you will not be able to upload/overwrite via WiFi if wifi_tools.lua or servernode.lua exceeds 1200 bytes. FTDI and COM ports still work if you have a wifi_tools.lua or servernode.lua file larger that 1200 bytes.

    There is no size limit on any other files.

    This is a fairly complicated Instructable. A lot of things have to happen just right for this to work. I have tested this on my setup with a number of different ESPs but I can't possibly foresee all combinations of variables.

    My setup is running Apache2 and PHP Version 5.4.on Linux. I believe linux based Apache2 systems are similar enough to not cause issues.

    Running on IIS or IOS will probably work, however, if it doesn't, look hard at line feeds and EOF handling. Possibly forward and backslashes in file addressing.

    Chrome, IE and Safari seem to be ok. Chrome/Safari on my Iphone have no problems. Yes, I update/configure my ESPs from my Iphone.

    I am connecting to my ESPs over a private LAN. I have not tested this over the Internet.

    I have posted all the source files on GitHub. Use them in a test environment!

    Of course, no guarantees!

    All source files here:

    GitHub sourse files

    Good luck managing your ESPs over WiFi!