Set Up an ESP8266 Automatic Update Server

Introduction: Set Up an ESP8266 Automatic Update Server

About: Programmer, Electronics tinkerer, lover of Arduino, ESP8266 and currently building an extensive home automation system for my remote 300-year old farmhouse in France

Many people are now using the ESP8266 in its many guises (ESP-01S, Wemos D1, NodeMCU, Sonoff etc) for home automation systems. If you write your own code (as I do) updating each of these separately even via OTA (over the air) becomes a bit tedious.

My own system, for example has 8x ESP-01S, 6x Wemos D1, 4x Sonoff Basic 12x Sonoff S20, 2x Sonoff SV and a NodeMCU which share a common code base, so that's 33 devices in all to update when I make a simple code change.

But there is an easier way: An "update server". The excellent Arduino IDE + ESP8266 core has a library to do most of the work (ESP8266httpUpdate), but you need to know how to set up your own server to make it work.

This Instructable shows you how using a NODE-RED server, but the same logic applies to any server technology of your choice e.g. Apache + PHP etc

Step 1: What You Need

  1. Arduino IDE
  2. ESP8266 core
  3. Any ESP8266 dev board with 1M or more flash RAM
  4. A Web Server (even a humble raspberry Pi will do - It's what I use)
  5. (optional) mkspiffs tool if you want to auto-update a SPIFFS file system image

Step 2: Create a Repository to Hold Binary Firmwares

On my server, I have a folder called /home/pi/trucFirmware which holds the various device firmwares and SPIFFS images

I maintain a separate binary for each hardware type (from a single source file with a few #defines) and when a new release is ready I use the Arduino IDE "sketch/Export compiled Binary" menu command for each target device.Note that even though there are 5 different hardware types, there are only two SPIFFS binaries: a 1M and a 4M version - constructed with the mkspiffs tool - since all the devices have either 1M or 4M flash.

Step 3: Create the Binaries

Using the Arduino IDE menu option sketch/Export Compiled Binary, create the firmware that will be uploaded to the device when it requests it from the update server.

If you need a SPIFFS binary you will need to install the mkspiffs tool.

Once you have it, building the SPIFFS binary is simple. I have a one-line batch file for the 1M version which takes the version number as a parameter (%1)

mkspiffs -c data/ spiffs_%1_1M.bin

and another for the 4M version:

mkspiffs -p 256 -b 8192 -s 0x0FB000 -c data/ spiffs_%1_4M.bin

I then copy all the compiled binaries and the SPIFFS .binary files over to the repository

Step 4: Create the Server Flow

I'm using NODE-RED, but the simple logic will be the same on any server technology / language.

a) Define a url which will listen for the ESP8266httpUpdate request. My raspberryPi serevr is on 192.168.1.4 and listens on port 1880 for /update with the hardware type appended. So if I'm going to request a binary for a Wemos D1 Mini, the url ends up as:

http://192.168.1.4:1880/update/d1_mini

b) Create code to handle the following logic:

ESP8266: "Hi, I'm running firmware version a.b.c, do you have a newer version?"
Server: "Let me see...ah yes I have a.b.d - here it comes..."

If a newer version does exists the server just sends it as a load of binary data in the http reply. The ESP8266httpUpdate class does the tricky part of copying the binary into memory, changing the firmware boot address to the new code than (if requested) rebooting the device to run the new code.

If on the other hand there is no higher version, it replies with a http 304 error which effectively says: "I have nothing for you" and your code continues to run as normal.

Step 5: Add the Server Logic

The first node in the flow "listens" for an http request to url http://192.168.1.4:1880/update with the device type appended. It passes this to "Construct search path" function node which has the following javascript code:

msg.type=msg.req.params.type;
var h=msg.req.headers; msg.version=h["x-esp8266-version"];

msg.mode=h["x-esp8266-mode"]; if(msg.mode=="sketch"){ msg.payload="/home/pi/trucFirmware/*.ino."+msg.type+".bin"; } else { var sz=h['x-esp8266-chip-size']; msg.payload="/home/pi/trucFirmware/spiffs_*_"+(sz/1048576)+"M.bin"; } return msg;

This just sets up the appropriate path with wildcard for the sys function which follows, which simply runs

ls - r

The output is then fed to the "Compare versions" function node:

var f=msg.payload.split("\n")[0];
msg.filename=f;

if(msg.mode=="sketch"){ f=f.replace("/home/pi/trucFirmware/truc_",""); f=f.replace(".ino."+msg.type+".bin",""); } else { f=f.replace("/home/pi/trucFirmware/spiffs_",""); f=f.replace(/_\dM\.bin/,""); }

if(msg.version < f){

node.warn("upgrade required"); node.warn("will return "+msg.filename); return msg; } node.warn("no upgrade"); msg.statusCode=304; msg.payload=[];

return msg;

The switch node then ensures that either the 304 "no update needed" message is sent or the actual new binary is returned and sent back to the device.

Step 6: Add Code to the Sketch to Request an Update

The sketch needs to have the following code included in it so that it will update automatically the next time you increase the version number:

#include<esp8266httpupdate.h>

... #define TRUC_VERSION "0_4_99"

#define SPIFFS_VERSION "0_5_0"

// THIS_DEVICE is set earlier depending on various compile-time defines // which eventually define the hw type, e.g. #define THIS_DEVICE "d1_mini" const char * updateUrl="http://192.168.1.4:1880/update/"THIS_DEVICE; // this is my raspberry Pi server, the 1880 is the default NODE-RED port // /update is the url I chose for the server to "listen" for, followed by the device type ... bool actualUpdate(bool sketch=false){ String msg; t_httpUpdate_return ret; ESPhttpUpdate.rebootOnUpdate(false); if(sketch){ ret=ESPhttpUpdate.update(updateUrl,TRUC_VERSION); // **************** This is the line that "does the business" } else { ret=ESPhttpUpdate.updateSpiffs(updateUrl,SPIFFS_VERSION); } if(ret!=HTTP_UPDATE_NO_UPDATES){ if(ret==HTTP_UPDATE_OK){

Serial.printf("UPDATE SUCCEEDED"); return true; } else { if(ret==HTTP_UPDATE_FAILED){

Serial.printf("Upgrade Failed"); } } } return false; }

Step 7: Finally, Initiate the Update

At boot time, or perhaps in response to an MQTT message (as I do) run the following code:

if(_actualUpdate(true)) ESP.restart();

// or for SPIFFS...

if(_actualUpdate(false)) ESP.restart();

The device will update itself and reboot running the latest code from the server. It's a lot simpler for me than manually updating 33 devices!

Lots more useful information about Home Automation, IOT and programming the ESP8266 can be found on My Blog

1 Person Made This Project!

Recommendations

  • Pocket-Sized Speed Challenge

    Pocket-Sized Speed Challenge
  • Audio Challenge 2020

    Audio Challenge 2020
  • Maps Challenge

    Maps Challenge

18 Discussions

0
mudassar.qm
mudassar.qm

Question 2 months ago on Step 7

the project is incomplete, can you please make a example from start to end. specially program of esp, where is to coded in main,Void etc or same as in micropyhton.

Thanks

0
sujataapatil
sujataapatil

4 months ago

Hello,

Thank you for this helpful instructable.
I am a beginner in Arduino & ESP & need help to implement this one.
Where shall I set the request headers "x-esp8266-version" & "x-esp8266-mode" in sketch to make this work?

0
vijayv66
vijayv66

1 year ago

IN my case I am getting magic byte error while uploading the binary file.
[HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK'
[HTTP-Client][handleHeaderResponse] RX: 'X-Powered-By: Express'
[HTTP-Client][handleHeaderResponse] RX: 'Access-Control-Allow-Origin: *'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 386432'
[HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/html; charset=utf-8'
[HTTP-Client][handleHeaderResponse] RX: 'ETag: W/"5e580-sbMqiSzVJszbBHAg13tl2+BG2to"'
[HTTP-Client][handleHeaderResponse] RX: 'Date: Tue, 19 Mar 2019 20:21:44 GMT'
[HTTP-Client][handleHeaderResponse] RX: 'Connection: close'
[HTTP-Client][handleHeaderResponse] RX: ''
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 386432
[httpUpdate] Header read fin.
[httpUpdate] Server header:
[httpUpdate] - code: 200
[httpUpdate] - len: 386432
[httpUpdate] ESP8266 info:
[httpUpdate] - free Space: 3866624
[httpUpdate] - current Sketch Size: 306208
[httpUpdate] - current version: 1
:rch 536, 536
:rch 1072, 536
:rch 1608, 536
:close
[httpUpdate] runUpdate flash...
:pd 4, 2144, 239
:rpi 297, 4
[httpUpdate] Magic header does not start with 0xE9
[HTTP-Client][end] still data in buffer (1905), clean up.


can some please help

0
LZolcsi
LZolcsi

Reply 8 months ago

I had the same issue and finally found the solution (as a matter of fact it is what @simonpierregagnon suggested):
You need to edit the File in node and change the Output mode to "single buffer object". This will sort out the problem.

0
simonpierregagnon
simonpierregagnon

Tip 10 months ago

Thanks for this guide! however after fiddling a couple of hours on this (I'm new on node-red) I feel like I could add some infos:

EDIT: Please ignore the following suggestion if you use non-integers in your firmware revision (i.e.: 3.2.4)

1.In the compare node at the conditionnal statement "if(msg.version < f)" I found out that we are comparing strings here, so inputting an inferior firmware revision file will lead to a firmware update. So I modified it to: "if(Number(msg.version) < Number(f))". This way we force integer convertion and evaluation.

2. It may seem obvious to some of you but I scratched my head a bit on this one...The node parameter "output" in the file node must be set at "a single buffer object" so that Httpupdate lib can parse it. Also, the field "filename" must be blank, path is passed on as a parameter.

3. Another not so obvious; the "switch node" can be an evaluation of the "msg.statusCode" variable so that it can take the first path if it's 304, and the second if it's anything else.

There you go, I hope this will help some people!

0
Phil Bowles
Phil Bowles

Reply 10 months ago

Forcing integer values is a seriously bad move and I absolutely would NOT recommend it.

99.9% of version numbers in the real world cannot be represented as a number and MUST be strings. What's the numeric conversion of "3.2.2beta" ? Forgetting the beta, whats the numeric conversion of 3.2.2 ? it makes no sense whatsoever.

It is up to YOU to arrange your versioning scheme so that this system works correctly, please don't confuse others by suggesting something that has not been thought through at all and simply will not work in the real world.

Please, all readers: IGNORE THIS SUGGESTION

0
simonpierregagnon
simonpierregagnon

Reply 10 months ago

I am willing to edit my tip for the benefit of the readers. I did some other testings without the integer conversion and everything seems fine. I am not sure how logical operators work with strings in this language and I feared some unwanted false positives might happen.

0
ahmed alkhabeer
ahmed alkhabeer

1 year ago

i have a couple of doubts
1. do i need to put the spiff part in my code or can I omit it
" ret=ESPhttpUpdate.updateSpiffs(updateUrl,SPIFFS_VERSION);" <<----- this part

0
scropion86
scropion86

1 year ago

Hi could you please share the JSON code for node-red part so we can replicate the idea

0
Phil Bowles
Phil Bowles

Reply 1 year ago

What "JSON code"? Do you mean the javascript source of the nodes? All the important ones are listed above in the body of the text.

0
JeremyC157
JeremyC157

2 years ago

Hello would it be possible for you to take a look at my instalation?

currently my life is returning to me

Upgrade Failed

I think the error comes from my configuration of the node server.

Thank you very much for your work

0
Phil Bowles
Phil Bowles

2 years ago

No, the clue is in the name of the library: ESP8266httpUpdate. It uses the http protocol, not ftp. PHP is a web.server feature, it has nothing to do with http. The easiest way to make the server is with node-red, it uses only a few lines of code. To be honest, I'm not really sure I understand your question exactly...

0
Phil Bowles
Phil Bowles

Reply 2 years ago

correction!!! php has nothing to do with FTP

0
JeremyC157
JeremyC157

Reply 2 years ago

good evening, I followed your tutorial for esp32 and I can not do the automatic update, your code and esp32 compatible?thks

0
Phil Bowles
Phil Bowles

Reply 2 years ago

I havent tried it on ESP32. Is there an ESP32 equivalent of the esp8266httpudate library? How are you initiating the update request? What webserver are you using and what code is running on it in response the the update request? What errors are you getting?

0
DIY Hacks and How Tos

Cool project. I think that is the most boards that I have seen stacked in one project before.

0
Phil Bowles
Phil Bowles

Reply 2 years ago

Thanks! I will be publishing more soon about all those different hardware devices that make up my own Home Automation system