This article documents the practical ruggedisation and onward development of an earlier Instructable : 'Pimping' your first IoT WiFi Device. Part 4 : IoT, Home Automation including all the necessary software functionality to enable the successful deployment in a domestic home environment.
As mentioned above this Instructable describes the bringing together of an earlier IoT example with a reliable systems design allowing for the successful handling of practical use cases such as; Catastrophic power loss, MQTT Broker failure, WiFi N/W failure, remote sensor reconfiguration, configurable reporting strategy to reduce network traffic and bespoke sensor calibration.
A total of 6 off devices were created (see pic 1 above) and distributed around my home to form my first IoT sensor network.
The Instructable also sees a review of the MQTT naming convention as used in the initial IoT Home Automation series giving way to a more balanced, practical structure allowing for simpler debugging of IoT traffic in a multi IoT device environment.
What follows are full design details of the IoT sensor including; construction, source code, testing strategy and OpenHAB configurations.
What parts do I need?
- 1 off ESP8266-01,
- 2 off 1uF Electrolytic Capacitors,
- 3 off 10K resistors,
- 1 off 330R resistor,
- 1 off 3mm dia. LED,
- 1 off LD1117-33v, 3v3 LDO VReg. (Farnell here),
- 1 off DHT22 Temperature/Humidity Sensor,
- 1 off Dual 4way 0.1" Connector,
- 1 off CAMDENBOSS RX2008/S-5 Plastic Enclosure, Potting Box, ABS, 38 mm, 23 mm (Farnell here),
- 1 off DC Power Connector, Plug, 1 A, 2 mm, Panel Mount (Farnell here),
- 1 off TO-220 Heatsink 24.4 °C/W (Farnell here),
- Various heat shrink tubing (yellow, Ebay here),
- Various lengths IDC ribbon cable,
- Heatsink compound,
- ESP8266-01 programming device. See here; Practical Circuit Construction with Strip Board, Step 9 onwards.
What software do I need?
- Arduino IDE 1.6.9
- Arduino IDE configured to programme the ESP8266-01. See here; Setting Up the Arduino IDE to Program the ESP8266-01
What tools do I need?
- Soldering Iron,
- Drill & various bits,
- Sturdy vice,
What skills do I need?
- A minimal grasp of electronics,
- Knowledge of Arduino and it's IDE,
- Rudimentary fabrication skills (soldering, hack-sawing, filing, drilling etc.),
- Some Patience,
- Some understanding of your home network.
- Circuit overview
- Software System Overview
- Software Overview
- Sensor Calibration
- MQTT Topic Naming Convention
- OpenHAB Configuration
- Testing the Design
- References Used
Step 1: Circuit Overview
Picture 1 above shows the full circuit design for the IoT sensor.
At the heart of the IoT device is the ESP8266-01 which is connected to a DHT22 temperature/humidity sensor via a 10K pull up resistor to GPIO2. An external 5v is sourced with a switched mode supply and fed to the device through a 2mm DC panel mount socket and regulated locally with an LD1117-33v, 3v3 LDO voltage regulator mounted to an external heat sink with a BZP M3 pan head screw and nut.
The design includes a 3mm red led connected to GPIO0 which is used to give local indication of the status of the IoT device during start up or any subsequent error condition. It can also be used to identify the device by manual activation via the openHAB interface.
The full design fits neatly into an ABS potting box as shown above in picture 2 and was laid out specifically to ensure the sensor is as far as possible from the regulator to prevent biasing due to local heating effects (picture 7 above).
The circuit board is a single piece of veroboard, cut to shape and made to fit into the enclosure (picture 3 above). This board is fixed in position with an M3 countersunk nylon screw and two nuts which fits flush with the underside of the sensor, thus allowing it to sit on a flat surface.
Pictures 4 ... 6 show various states of construction.
Step 2: Software System Overview
This IoT temperature and humidity sensing device contains six key software components as shown in pic 1 above.
This is the on-board SPI Flash Filing System and is used to hold the following information (see pic 2 above);
- Icons and 'Sensor Configuration Home Page' html: Served up by the IoT device when it is unable to connect to your IoT WiFi network (usually due to incorrect security information) and provides the user with a means of remotely configuring the sensor without the need to re-programme or upload new SPIFFS content.
- Security Information: This holds the information used at power up by the IoT device to connect to your IoT WiFi network and MQTT Broker. Information submitted via the 'Sensor Configuration Home Page' is written to this file ('secvals.txt').
- Calibration Information: The information contained within this file ('calvals.txt') is used to calibrate the on-board temperature/humidity sensor should it be necessary. Calibration constants can only be written to the IoT device via MQTT commands from an MQTT broker.
Note : To initially set up the device see here for full details on how to use SPIFFS with the Arduino IDE.
This functionality is invoked when the IoT device has failed to connect to your WiFi network as a WiFi station and instead has become a WiFi access point something akin to a domestic WiFi router. In the case of such a router you would typically connect to it by entering the IP Address of something like 192.168.1.1 (usually printed on a label affixed to the box) directly into your browser URL bar whereupon you would receive a login page to enter the username and password to allow you to configure the device.
For the ESP8266 in AP mode (Access Point mode) the device defaults to the IP address 192.168.4.1, however with the mDNS server running you only have to enter the human friendly name 'SENSORSVR.local' into the browser URL bar to see the 'Sensor Configuration Home Page'.
The MQTT client provides all the necessary functionality to; connect to your IoT network MQTT broker, subscribe to the topics of your choice and publish payloads to a given topic. In short it provisions IoT core functionality.
HTTP Web Server
As mentioned above, if the IoT device is unable to connect to the WiFi network whose SSID, P/W etc. is defined in the Security Information file held in SPIFFS the device will become an Access Point. Once connected to the WiFi network provided by the Access Point, the presence of an HTTP Web Server allows you to directly connect to the device and change it's configuration via the use of an HTTP Web Browser it's purpose being to serve up the 'Sensor Configuration Home Page' web page which is also held in SPIFFS.
This functionality gives the IoT device the capability to connect to a domestic WiFi network using the parameters in the Security Information file, without this your IoT device will not be able to subscribe/publish to the MQTT Broker
WiFi Access Point
The ability to become a WiFi Access Point is a means by which the IoT device allows you to connect to it and make configuration changes via a WiFi station and a browser (such as Safari on the Apple iPad).
This access point broadcasts an SSID = "SENSOR" + the last 6 digits of the MAC address of the IoT device. The password for this closed network is imaginatively named 'PASSWORD'
Step 3: Software Overview
To successfully compile this source code you will need the following extra libraries;
- By : Nick O'Leary
- Purpose : Enables the device to publish or subscribe to MQTT topics with a given Broker
- From : https://github.com/knolleary/pubsubclient
- By : Adafruit
- Purpose : Library for DHT Temperature/Humidity Sensor
- From : https://github.com/adafruit/DHT-sensor-library
The software makes use of the state-machine as shown in pic 1 above (full copy of source given below). There are 5 main states as below;
- This initialisation state is the first state entered after power up.
- This state is entered if after power up an invalid or missing secvals.txt file is detected
- PENDING NW
- This state is transitory, entered whilst there exists no WiFi network connection
- PENDING MQTT
- This state is transitory, entered after a WiFi network connection has been made and whilst there exists no connection to an MQTT broker on that network.
- This is the normal operational state entered once both a WiFi network connection and an MQTT Broker connection has been established. It is during this state the temperature and humidity functionality of the sensor is published to the MQTT Broker.
The events controlling transitions between states are described in pic 1 above. Transitions between states is also governed by the following SecVals parameters;
- 1st MQTT Broker IP Address. In dotted decimal form AAA.BBB.CCC.DDD
- 2nd MQTT Broker Port. In Integer form.
- 3rd MQTT Broker connection attempts to make before switching from STA mode to AP mode. In Integer form.
- 4th WiFi Network SSID. In free form text.
- 5th WiFi Network Password. In free form text.
As mentioned above if the IoT device is unable to connect as a WiFi Station to the WiFi network who's SSID and P/W is defined in secvals.txt held in SPIFFS the IoT device will become an Access Point. Once connected to this access point it will serve up the 'Sensor Configuration Home Page' as shown above in Pic 2 (by entering either 'SENSORSVR.local' or 192.168.4.1 into your browsers URL address bar). This home page allows the reconfiguration of the sensor via an HTTP browser.
Remote Access whilst in the ACTIVE state
Once connected to the MQTT Broker it is also possible to both re-calibrate and reconfigure the device via MQTT topic publications. The file calvals.txt has R/W access and secvals.txt has write only access exposed.
During the boot sequence the IoT device led gives the following debug feedback
- 1 Short flash : No Config file located in SPIFFS (secvals.txt)
- 2 Short flashes : IoT device is attempting to connect to WiFi network
- Continuous illumination : IoT device is attempting to connect to MQTT Broker
- Off : Device is active
- Note 1 : The 'Sensor Configuration Home Page' does not use secure sockets and therefore relies on your network being secure.
- Note 2 : In order to programme each IoT device the MQTT string will require editing prior to downloading. This is because the number of the sensor has been embedded into the MQTT topic string. ie. 'WFD/THSen/100/HumdStatus/1' for my 6 devices they are numbered 1...6 respectively.
Step 4: Sensor Calibration
When the IoT device powers up, as part of the boot sequence a file named 'cavals.txt' is read from SPIFFS. The contents of this file are calibration constants as indicated above in pic 1. These calibration constants are used to adjust the readings acquired from the sensor to bring them into line with a reference device. There is one further value which defines a reporting strategy for the device and is described below along with the procedure followed to calibrate the sensors.
This parameter determines how the remote sensor reports any ambient parametric changes local to it. If a value of 0 is selected the remote sensor will publish any change it sees in the temperature or humidity values each time the sensor is read (approx every 10 seconds). Any other value will delay the publication of a change by 1...60 minutes. Modifying this parameter allows for optimisation of MQTT network traffic.
To calibrate the sensors they were placed in close physical proximity with each other as show above in pic 2. Along side them I placed a DMM with a calibrated thermocouple attached (Fluke 87 V) and then monitored the outputs from each device via the OpenHAB temperature trend page over the course of a day to get a good temperature swing. I noted both the static offset (elevated zero 'C') and the rate of change of each device (gain, or slope of graph 'M') relative to that of the value coming from the calibrated thermocouple. I then calculated the simple y=mx+c relationship (I found it was sufficiently linear as to be a close approximation to a straight line graph) and programmed any necessary corrections into the calibration constants via MQTTSpy.
The devices were then monitored for a further 24hrs to ensure the calibration was successful. An indication of which was the temperature traces on the OpenHAB temperature trend page were all pretty much on top of each other.
Of course if you are only interested in an approximation to the temperature you can leave all the calibration values as default.
As I possess no means to accurately record or even control local ambient humidity, to calibrate the sensors I used a similar approach that of the above, by placing all the devices in close physical proximity (pic 2) and simply monitoring their output via the OpenHAB Humidity tend page. I then picked device #1 as the calibration reference and calibrated all the devices relative to this.
Step 5: MQTT Topic Naming Convention
After much trial and error I settled on the topic naming convention outlined in pic 1 above.
It's not perfect but it does allow for useful filters to be applied to see all sensor outputs for a given parametric value thus allowing for easy comparison as in pic 2 above with MQTTSpy. It also supports reasonably extensible logical groupings of functionality within a given IoT device.
In implementing these topics in software I used hard coded topic strings with fixed, embedded numerical identifiers for each device as opposed to dynamically generating the topics at run time so as to save on RAM and keep performance high.
Note : If you're not sure how to use MQTTSpy see here 'Setting Up an MQTT Broker. Part 2 : IoT, Home Automation'
Step 6: OpenHAB Configuration
I modified the OpenHAB configuration given in my earlier Instructable (here) and added in individual entries for;
- Living Room,
- Guest Bedroom
- Master Bedroom
In the site map see pic 1 above.
For each of these entries I added individual sitemaps exposing local ambient values (See pic 2 above);
- Heat Index
I also included a switch to control the local led mounted within the sensor.
Pictures 3 ...5 show individual live traces over the period of 24hrs for temperature, humidity and RSSI (Received Signal Strength Indication, basically a measure of how well the sensor can see the WiFi network).
Picture 6 gives an example of a long term humidity trend over the period of a week.
Note 1 : If you're not sure how to use OpenHAB see here 'Setting Up and Configuring OpenHAB. Part 6 : IoT, Home Automation'
Note 2 : A copy of the modified sitemap, rules and items files, Icons etc. is given below.
Step 7: Testing the Design
For the most part I tested the IoT device over the MQTT connection with MQTT Spy, monitoring led output and debug traffic on the serial interface. This allowed me to exercise all of the available subscribed topics and check the published responses. Although this was achieved manually and become a bit tedious at times, it enabled 100% coverage.
However the main state machine proved to be a little tricky to test as it relied upon the presence or absence of a WiFi network, access to which requiring specific parameter sets. It simply wasn't practical to use the home network for this.
To get round this issue I created my own set of dummy networks using ESP8266-01 configured as Access Points (pic 1) with SSIDs of 'DummyNet1' and 'DummyNet2' respectively. Using the circuit in pic 2 above the led gave an indication if an IoT device had connected to it. Whilst this wasn't a perfect testing solution (ie. each of these dummy WiFi networks didn't contain an MQTT server) it was possible to fully test the state machine.
I've included a copy of the source code below.
Step 8: Conclusion
The software in the IoT devices has worked reliably for many months now recovering from household power outages (mainly caused by myself). Overall they are pretty robust devices giving consistent and accurate data.
In developing software routines to read and write to SPIFFS I wrote code which in hind sight may be a little more advanced than I had intended, using void pointers, recasting and pointers to pointers. Whilst it's very flexible and does the job well, next time I may use JSON someting along the lines of ConfigFile.ino to keep it a little simpler.
- Arduino GIT HUB Core
- ConfigFile.ino Source
I had intended to use an mDNS client to connect to the Broker but the library was too flakey. This is why it is necessary to specify the MQTT Broker IP address as opposed to 'MQTTSVR.local'. Should the mDNS library become more stable in the future I'll add this capability to the device.
It would have been nice to have a means of both accurately monitoring and control ambient humidity to calibrate the sensors against. However that said the calibration method chosen gives good relative readings and seems reasonably accurate in line with the specification in the DHT22 data sheet.
Finally, given the complexity of the software I found fully testing the code after a major change was becoming time consuming. I may consider automated testing at a later date.
Step 9: References Used
I used the following sources to put this Instructable together;
- By : Nick O'Leary
- From : https://github.com/knolleary/pubsubclient
- By : Adafruit
- From : https://github.com/adafruit/DHT-sensor-library
Participated in the
Sensors Contest 2017