ESP8266, Weather Station MQTT Agent; Raspberry Pi IOT Server

This document is not the last version there have been some updates, mainly in mqtt reader (I have eliminated node-red and replaced by an ad-hoc program).

The latest version is located in

https://github.com/churruscat/weather_station

Step 1: 1. Introduction

The goal was to set up a weather station without using IOT services from any cloud provider. Besides the learning challenge, it provides independency from internet communications and IT providers.

We will use a Raspberry as the IOT Server and ESP8266 with their correspondent gauges as clients. As you separate clients and server, you can have as many clients as you want, measuring data in different places, or polling for different data (e.g. a rain gauge only makes sense outside ;-) ) depending on where the client is installed.

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It was designed as an extremely lightweight publish/subscribe messaging transport. It is useful for connections with remote locations where a small code footprint is required and/or network bandwidth is at a premium. For example, it has been used in sensors communicating to a broker via satellite link, over occasional dial-up connections with healthcare providers, and in a range of home

MQTT is a very convenient protocol to implement IOT; it's simple, reliable and light. All (or almost all) IOT cloud providers implement it and provide an MQTT server. In this case, I am going to be provider independent and connect to a local MQTT server, but the agent will be the same, with only minor changes in server address and authentication data, as userID and password.

The ESP12 (aka NodeMCU ESP8266) is a very cheap circuit that can be found for around 2€. It is an Arduino-like circuit with integrated WIFI and a USB port. There are some versions of the circuit. Even though V3 has the higher version number and hence it is supposed to be the most advanced, I would try to avoid them versions, they use a bigger footprint and have no clear advantages. I rather prefer v2; I buy them in AliExpress at a very low price and, so far, I have not had any problem. Should you want more info about these chips, there is lot of info in internet, e.g. https://frightanic.com/iot/comparison-of-esp8266-nodemcu-development-boards/

I program ESP8266 chips using Arduino IDE. The language used is C/C++ with some small limitations, but powerful enough. Language reference can be found at Arduino official page https://www.arduino.cc/reference/en/, but there is also plenty of information, forums and ideas in other popular pages.

Hence, you have a chip with multiple pins to connect sensors (Digital, analog, I2C, SPI, MISO ...), programable in C, with USB interface and WIFI in a 25mmx48mm (1"x1.9") circuit. This means that it's possible to connect multiple sensors of any kind (I have to say that I had some problems trying to use two UART interfaces simultaneously), to control them and to communicate with the server with WIFI. As there are libraries for almost every sensor and lots of them for networking and subsystems -e.g. not only TCP/IP or MQTT (see below) but also HTTP, (Web)sockets, SSL and many more- it is incredible easy to build complex systems.

This project implements sensors for a weather station, but once everything is set up, it is easy to evolve to other projects with different types of sensors.

So, let us begin.

First things first. We must install and configure Arduino IDE to edit, compile and upload programs to our ESP8266 device.

1.Install and configure Arduino IDE

1.1. First, install the program.
Download form from https://www.arduino.cc/en/Main/Software , install the IDE and the drivers that come with it. LINUX: If you are using Linux and prefer using built-in package manager (e.g. aptitude), check whether your package manager has the latest version available. MAC: Install also these drivers https://www.arduino.cc/en/Main/Software

1.2. Add ESP8266 board to Arduino IDE, by adding the board manager link to the preferences Start Arduino IDE and open “Preferences” menu in File -> Preferences Check the boxes in the preferences according to the picture below for a better development interface.

Step 2: Arduino IDE Installation

Also, add the link http://arduino.esp8266.com/stable/package_esp8266com_index.json to “Additional Boards Manager URLs” and save settings by pressing ok.

Restart Arduino IDE so it downloads board libraries upon starting.

1.3. Install the new board manager for ESP8266 chip

In the Arduino IDE main screen select Tools -> Board (…) -> Boards manager…

Search for “ESP8266”, then click the suggested result and press install

1.4. Confirm the installation and choose the NodeMCU board for the current board

In Arduino main screen select Tools -> Board (…) and then select “NodeMCU 1.0 (ESP 12-E module)” from the list

1.5 We are done!

If you are using Linux, ensure that your user have permissions to use the ports. Instructions are available at www.arduino.cc.

Windows (also in MAC?): If the chip is not seen by Arduino IDE in “Tools ->port”, or there are communication problems, check that you have proper USB-to-Serial connection drivers installed in your system. We can find your chip’s Serial type from the bottom of the chip. In most cases, it should be either CP210x or CH340(G). Both drivers are found in their manufacturers page, easy to find and download.

Now, we can begin to prepare our meteo gauge.

2. Sensors

We will set up gauges for temperature, air humidity, air pressure and rain.
Even though it is not a meteorological measure, we will also connect a soil moisture sensor.

2.1. Temperature, air humidity and atmospheric pressure. BME280.

There are sensors to read each of the different magnitudes, but, using a single sensor for the three makes everything easier: single wiring and only one library needed. That is why I prefer to use BME280.

BME280 sensor

There are many circuits prepared for this sensor built by Adafruit, Sparkfun or Chinese manufacturers. All of them are very cheap (about 2€) and quite similar. The chip we are using here provides an I2C interface. Its address is documented as 0x77 in many places but, after some headaches and reviewing cabling, soldering ...etc., I found that in the ones I bought it is 0x76. To read sensor data, we will also need a library, I use to the Adafruit_BME280 library (code on github). It is also available from the Arduino library manager and is the method I recommend, so it will notify when there are upgrades.

From the Arduino IDE open up the library manager...

And type in Adafruit bme280 to locate the library. Click Install

If not installed, you will also need to install the wire library, that is also available in Arduino Library Manager. So look for ‘wire’ an install the library.

2.2. Soil Moisture

Thera are also cheap soil moisture gauges but, there are not especially accurate. Cheap models (below 3€) are of two types:

resistive where the sensor measures the electric resistance of the soil
capacitive, where the sensor measures the electric capacity of the soil surrounding the sensor

Both are subject to corrosion, much more in the case of the resistive ones. Both type of sensors give a value depending on how moist is the soil, but there are many other variables that impact the measure (Ph, compaction, composition …). Even though the read value depends on the type of soil, it is relative to moisture, so we will need to take some measures- from completely dry to soaked – in order to know what the scale limits for that particular soil are.

To limit corrosion on the sensor, we will only power it when needed so there only will be tension when we want to read a value. What we will do is putting a transistor (a 2N222 is a suitable one) that will connect and disconnect power using the transistor base (see circuit and code in following lines).

2.3. Rain gauge

The rain gauge I use is another sensor that does not need special libraries, only analog and interrupt counter for the rain gauge. Interrupt handling is a bit trickier (we will cover it below).

https://www.amazon.co.uk/MISOL-Spare-weather-stati...

To complete the sensors used (I did not put an anemometer, because at my home is senseless, but its mechanism is very similar to this one), we will set up a rain gauge. To use an automatic rain gauge, one common solution is a tipping bucket rain gauge. It consists in a little seesaw with two buckets. the rain fills up a bucket on one end, so it tips over (and empties) and the bucket on the other side starts to fill. Each time the bucket tips, it makes a momentary electrical connection, using a reed switch or a hall switch. The buckets are calibrated to a volume of water, which means if you count how many times the switch closes, multiply it by the bucket volume and divide by the area of the funnel that collects the water, you will know how much rainfall there has been.

In Amazon you can find rain gauges at very affordable prices. I used https://www.amazon.co.uk/MISOL-Spare-weather-stat... but surely it is sold in other Web pages. This one is calibrated so every tip represents 0.2794 mm (to translate to inches, 0.011”, obtained by dividing this number by 2.54), it has a RJ11 plug at the end of the wire, you will have to cut it to connect to your ESP8266.

2.4. Circuit. Connecting the components

The connections of the three sensors are independent, so you can only use the ones you need. In the program, it is easy to remove those that you are not using.

Step 3: Arduino Program. Sensor Module

Now, you have the environment prepared and the chip connected, so it is
time to run the program that will read data from the sensors.

First, we will define a macro to expand printing in the Arduino IDE environment in debug mode, so when everything is working you can just turn a variable (CON_DEBUG) off and make the program lighter.

<pre>#define CON_DEBUG
#ifdef CON_DEBUG<br>  #define DPRINT(...)    Serial.print(__VA_ARGS__)
  #define DPRINTLN(...)  Serial.println(__VA_ARGS__)
#else
  #define DPRINT(...)     //Blank line
  #define DPRINTLN(...)   //  Blank line
#endif

This macro will expand to Serial.print if CON_DEBUG is defined, and to nothing if it is not. It is quite convenient because it makes unnecessary to put #ifdef all along the code or to make a call with direct return when DEBUG is off as other coding techniques do.

3.1. BME280

So, let us start with the program. We have to include the needed libraries Wire.h, Adafruit_Sensor.h and Adafruit_BME280.h for reading BME280 data; take into account that it is an I2C device and it is a bit more complicated. File PIN_NodeMCU.h describes the reltionship between PIN number and IO number. Also, I set an altitude pressure correction: PRESSURE_CORRECTION. My home is at 647 m altitude, to set your correction depending on your sensor altitude, there are many pages that convert barometric pressure depending on altitude (look for ‘barometric altitude’ and you will find many of them), just calculate the corrected/read factor and you have it. Other point that can be frustrating is I2C address; in some documentation it says that the I2C address for the BME280 is 0x77, but all the sensors I bought have 0x76. Try with both addresses before struggling with wires and soldering (as I did ☹). Once BME280 is correctly wired and using the provided libraries, it is quite easy to read data from it.

<pre>#include <PinNodeMCU.h>        // description of PINs
#define SDA D5                         // for BME280 I2C 
#define SCL D6                         // SCL PIN
#include   <Wire.h>                   // libraries for I2C
#include <Adafruit_Sensor.h>    // libraries for BME280
#include <Adafruit_BME280.h> 
#define PRESSURE_CORRECTION (1.080)               // HPAo/HPHh at 647m
#define BME280_ADDRESS   (0x76)                         //IMPORTANT, sometimes it is 0x77<br>

Once libraries are included and
constants defined, it is quite easy to read data:

<pre>float bufTemp,,bufHumedad,,bufPresion;     //variables used for reading
  bufHumedad= sensorBME280.readHumidity();   
  bufTemp= sensorBME280.readTemperature(); 
  bufPresion=sensorBME280.readPressure()/100.0F*PRESSURE_CORRECTION;                //HectoPascals

3.2. Soil moisture

Also, reading the moisture sensor is quite simple, the only thing that must be remembered is that we are using a transistor to power on and off the sensor. So, we need to:

<pre>/* activate soil sensor<br>setting the transistor base */
  digitalWrite(CONTROL_HUMEDAD, HIGH); //activate moisture reading
  espera(10000);                                                   //wait to stabilize (maybe less than 10 sec is enough)
  humedadCrudo = analogRead(sensorPin); // and read soil moisture
  digitalWrite(CONTROL_HUMEDAD, LOW);  // disconnect soil sensor

3.3. Rain gauge

A bit trickier is reading the pluviometer. To read the number of times the tip has moved, we are going to use interrupts.

First of all, we attach and interrupt to a PIN,

<pre>#define interruptPin D7 //PIN where I'll connect the rain gauge
pinMode(interruptPin,INPUT);

Then, let’s define the function that will be called whenever an interrupt happens (i.e. whenever hall sensor detects a change). This function must be of a special type, ICACHE_RAM_ATTR, and as simple as possible, because the ESP8266 will only run the interrupt function above anything else.

<pre>// Interrupt counter for rain gauge<br>void ICACHE_RAM_ATTR balanceoPluviometro() {   
  contadorPluvi++;
}

Now, we connect the interrupt function so it is called whenever signal in interruptPIN goes from low to high;

attachInterrupt(digitalPinToInterrupt(interruptPin),
balanceoPluviometro, RISING);

So, contadorPluvi is incremented by one each time interruptPIN goes from low to high, i.e. each time the rain gauge tips.

To read rain, we multiply the number of tips by the volume (mm, equal to l/m2 )

<pre>lluvia+=contadorPluvi*L_POR_BALANCEO;<br>//to set counter to zero, we must detach the interrupt (and then, attach it back)
detachInterrupt(digitalPinToInterrupt(interruptPin));
contadorPluvi=0;
attachInterrupt(digitalPinToInterrupt(interruptPin),
balanceoPluviometro, RISING);

3.4. Reading data (Putting all together).

Now, it is time to put all together. To read data, I merged everything a a function called tomaDatos() that reads all three sensors. To make things simpler, I use some global variables to store the data read. So, when tomaDatos() is called, five global variables are set. I know that this is not a best practice, but I was lazy about setting a structure, maybe I will correct it in next version. To mitigate random deviations in the data we read, I read twice and calculate the average

<pre>#define SDA D5    // for BME280 I2C <br>#define SCL D6     // SCL PIN
#define interruptPin D7  // PIN where I'll connect the rain gauge
#define sensorPin    A0   // Soil humidity sensor analog PIN
#define CONTROL_HUMEDAD D2   // Transistor base that switches on and off soil sensor
#define L_POR_BALANCEO 0.2794 // liter/m2 (=mm) for every rain gauge interrupt

#include<wire.h>  <wire.h> // libraries for I2C
</wire.h><wire.h>#include <<adafruit_sensor.h>Adafruit_Sensor.h>  </adafruit_sensor.h><adafruit_sensor.h>// libraries for BME280
</adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h>#include <Adafruit_BME280.h>
</adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h>#include <PINNodeMCU.h></adafruit_bme280.h></adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h><pin_nodemcu.h>   // description of PINs
</pin_nodemcu.h></adafruit_bme280.h></adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h><pin_nodemcu.h>#define PRESSURE_CORRECTION (1.080) // HPAo/HPHh at 647m
</pin_nodemcu.h></adafruit_bme280.h></adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h><pin_nodemcu.h>#define BME280_ADDRESS   (0x76)   //IMPORTANT, sometimes it is 0x77
</pin_nodemcu.h></adafruit_bme280.h></adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h><pin_nodemcu.h>volatile int contadorPluvi = 0; // must be 'volatile', for counting interrupt. Counts rain gauge tips
</pin_nodemcu.h></adafruit_bme280.h></adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h><pin_nodemcu.h>// ********* these are the sensor variables that will be exposed ********** 
</pin_nodemcu.h></adafruit_bme280.h></adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h><pin_nodemcu.h>Adafruit_BME280 sensorBME280;   // this represents the BME280 sensor
</pin_nodemcu.h></adafruit_bme280.h></adafruit_sensor.h></wire.h><wire.h><adafruit_sensor.h><adafruit_bme280.h><pin_nodemcu.h>float temperatura,humedadAire,presionHPa,lluvia=0,sensacion=20;
</pin_nodemcu.h></adafruit_bme280.h></adafruit_sensor.h></wire.h>int humedadMin,humedadMax,humedadSuelo,humedadCrudo;
int humedadCrudo1, humedadCrudo2;
// Interrupt counter for rain gauge
void ICACHE_RAM_ATTR balanceoPluviometro() {  
  contadorPluvi++;
}
/* get data function. Read the sensors and set values in global variables */
boolean tomaDatos (){
  float bufTemp,bufTemp1,bufHumedad,bufHumedad1,bufPresion,bufPresion1;
  boolean escorrecto=true;  //return value will be true unless there is a problem
  /* read and then get the mean */
  bufHumedad= sensorBME280.readHumidity();   
  bufTemp= sensorBME280.readTemperature();
  bufPresion=sensorBME280.readPressure()/100.0F;
  /* activate soil sensor setting the transistor base */
  digitalWrite(CONTROL_HUMEDAD, HIGH);
  espera(10000);  
  humedadCrudo = analogRead(sensorPin); // and read soil moisture
  humedadCrudo=constrain(humedadCrudo,humedadMin,humedadMax); 
  digitalWrite(CONTROL_HUMEDAD, LOW);  // disconnect soil sensor
  // calculate the moving average of soil humidity of last three values 
  humedadCrudo=(humedadCrudo1+humedadCrudo2+humedadCrudo)/3;
  humedadCrudo2=humedadCrudo1;
  humedadCrudo1=humedadCrudo;
  // read again from BME280 sensor
  bufHumedad1= sensorBME280.readHumidity();
  bufTemp1= sensorBME280.readTemperature();
  bufPresion1= sensorBME280.readPressure()/100.0F;
  DPRINTLN("Data read"); 
  lluvia+=contadorPluvi*L_POR_BALANCEO;
  detachInterrupt(digitalPinToInterrupt(interruptPin));
  contadorPluvi=0;
  attachInterrupt(digitalPinToInterrupt(interruptPin), balanceoPluviometro, RISING);
  if (humedadMin==humedadMax) humedadMax+=1; 
  humedadSuelo = map(humedadCrudo, humedadMin, humedadMax, 0, 100);
  /* if data could not be read for whatever reason, raise a message (in CONDEBUG mode) 
    Else calculate the mean */
  if (isnan(bufHumedad) || isnan(bufTemp) || isnan(bufHumedad1) || isnan(bufTemp1) ) {       
     DPRINTLN("I could not read from BME280msensor!");       
     escorrecto=false;  // flag that BME280 could not read
  } else {
  temperatura=(bufTemp+bufTemp1)/2;
  humedadAire=(bufHumedad+bufHumedad1)/2;
  presionHPa=(bufPresion+bufPresion1)/2*PRESSURE_CORRECTION;
  if (temperatura>60) escorrecto=false;  //if temperature out of reasonable range
  if ((humedadAire>101)||(humedadAire<0)) escorrecto=false;    // or humidity
  DPRINT("\tTemperature: \t ") ; DPRINT(temperatura);
  DPRINT("\tAir humidity: \t "); DPRINT(humedadAire);
  DPRINT("\tPressure HPa : \t "); DPRINT(presionHPa);
  DPRINT("\tMoisture: \t ")     ; DPRINT(humedadSuelo);
  DPRINT("\tRaw Moisture: \t"); DPRINTLN(humedadCrudo);  
  } 
  return escorrecto;
}

3.5. Sending data

Once data is read, it must be sent somewhere where it can be stored, manipulated and visualized. The solution selected is to send data to an mqtt broker. This broker can be hosted in a cloud provider (e.g. IBM bluemix, or AWS). In my case, I implemented it in a Raspberry, as described in “XXXXXXXXXX”. To send data, we will use the function ‘bool enviaDatos(char * topic, char * JsonString)’, defined in another file (see ‘networking module’). This function sends a JSon string.to an MQTT server. This is done with publicaDatos()

<pre>/* this function sends data<br>to MQTT broker */
void publicaDatos() {
  int k=0;
  char signo;
  boolean pubresult=true;  
  while(!tomaDatos()) {   // if tomaDatos() returns false, retry 30
times 
     espera(1000);        // waiting 1 sec between iterations
     if(k++>30) {         // after 30 iterations with no data,
return
      return; 
     }
  }
  // Data is read an stored in global var.
Prepare data in JSON mode
  if (temperatura<0) {  // to avoid problems with sign
    signo='-';          // if negative , set '-' character
    temperatura*=-1;  // if temp was negative, convert it positive 
  }  else signo=' ';
  // format a string to prepare the message
sprintf(datosJson,"[{\"temp\":%c%d.%1d,\"airH\":%d,\"moisture\":%d,\"moitsRaw\":%d,\"HPa\":%d,\"mm\":%d.%1d},{\"deviceId\":\"%s\"}]",signo,(int)temperatura,(int)(temperatura * 10.0) % 10,\
        (int)humedadAire, (int)humedadSuelo,(int)humedadCrudo,(int)presionHPa,
        (int)lluvia, (int)(lluvia * 10.0) %10,DEVICE_ID);
  // and publish them.
  pubresult =enviaDatos(publishTopic,datosJson);
  if (pubresult) 
    {lluvia=0;}      // I sent data was successful, set rain to zero 
}

To wait between measurements, we will also build a function that keeps other important ESP8266 routines (e.g. those related to WiFi). I wrote espera(Ulong waitMilliseconds), that is a simple function that waits n milliseconds while maintaining internal routines.

Step 4: Arduino Program Files

To make things easier, I have grouped this code in four files:

ESP12_BME280.ino : Main code -I mean, setup() and loop() functions- and sensor reading functions
1stDevice.h : Definitions for each device (there can be several devices reading sensors in different places).
mqtt_mosquitto.ino. networking code, setting up and connecting WiFi, initializing mqtt client and functions for sending data to the mqtt broker.
Mqtt_mosquitto.h: definitons for WiFi (SSID, password) and mqtt broker (address, pasword). We have an agent that sends data… now, we need a server

Step 5: Setting Up the Docker Based Raspberry IOT Server

The idea is to set up an IOT server in a Raspberry. The requirements for such a system are:

It must be able to receive, process and store MQTT messages,
It must provide a tool to program what to do with those messages
It must provide a way to store the data received.
In addition, It will also provide an interface to graphically show the stored data

And a way to access and manage the server components from internet.

The first thing I did was to select the different components, the limitations to this selection are: they must be open source, easy to manage and can run in a container in a Raspberry.

Another requirement was that every component could be set up as a container. I decided to install each single component in a separate container because it is an easy way to give independency to each subsystem. Each component runs in a separate container, so it is possible to start, stop, reconfigure, update re-install etc. without interfering the others. It also facilitates migrating from Raspberry to cloud. Containers are transportable easily (may be not directly) from one infrastructure to a quite different one. To simplify container usage, I will use docker as the container management software as it is widely implemented, easy to use and really robust.

With these requirements, and after some research, I selected the following ones:

Mosquitto as MQTT Broker. Mosquitto is widely used, stable, easy to configure and gives all the features I needed (and I likely will need in the future).
Node-red. This program is an easy way to link MQTT queues, devices and software components with a flow editor. Really friendly to use, scalable in functionality, and there are many blocks already developed but not limited to existing elements, as you can program easily your own block to adapt to your particular needs.
Influxdb. This database is incredible fast, simple and specially adapted to time series data. Has some features like policies and aggregations that are really interesting. Furthermore, it is very easy to query, accessible with CURL and also there is a Python module that eases the integration of stored data with other applications.
Grafana. A tool really easy to use. Preparing graphics to visualize stored data is a child's play. It provides many formats and timescales can be changed interactively. Also, it integrates smoothly with influxdb.
Nginx. Nginx is a very complete piece of software that can be used to many things related to route, balance and mask workload. In this case, I use it only as a reverse proxy, but could easily be scaled as a load balancer.
Telegraf. In addition, I installed telegraf to monitor the Raspberry itself. I store load data in influxdb and view it graphically with Grafana. Incredibly simple to install, configure and maintain.

We will also need some kind of Name server and certificate managing.I will also use docker-compose to simplify image management dependencies and parameters used to start each container.

So .... let's start.

1. Installing Docker

First of all, as usual, be sure that your Raspberry is at latest level. As sometimes the process is long, I run update and upgrade in the same line with 'y' (yes) answered to all prompts:

sudo apt-get update -y && sudo apt-get upgrade -y

It is also advisable to set a fixed IP address; to do so, edit /etc/dhcpcd.conf and add the lines (for an ethernet interface):

<pre>interface eth0 static
ip_address=aa.bb.cc.dd.ee/24
static routers=rr.ss.tt.uu
static domain_name_servers=nn.mm.oo.pp qq.rr.ss.tt
sudo reboot now

Now, install docker; first the necessary (transport) packages:

sudo apt install -y  apt-transport-https  ca-certificates software-properties-common

and then add docker DPG key

 curl -fsSL <a href="https://download.docker.com/linux/debian/gpg"> <a href="https://download.docker.com/linux/debian/gpg<font"> https://download.docker.com/linux/debian/gpg<font...< a=""> color="#001000"> </font></a> sudo apt-key add -</font...<></a>

and Docker repository

 echo "deb [arch=armhf] <a href="https://download.docker.com/linux/debian"> <a href="https://download.docker.com/linux/debian"> https://download.docker.com/linux/debian </a> </a>\$(lsb_release -cs) stable" | \<br><p>       sudo tee
/etc/apt/sources.list.d/docker.list</p><strong></strong><u></u><sub></sub><sup></sup><del></del>

As there is a new repository, we need an update

<pre>sudo apt update<br> sudo apt install -y docker-ce

Finally, add your own user (this is Pi as default user or the one you created if you did it) to docker group

          sudo usermod -aG docker MYUSERID

Starting, stopping and updating docker images is tedious and sometime requires long commands. To simplify these tasks and to manage dependencies I recommend using docker-compose. It helps to control the behavior of your container using a plain text (yaml) file. Docker-compose is a tool written in python, so the easiest way to install it is with pip (it is the Python Package Management System), so we begin installing Python pip:

sudo apt-get -y install python-pip

once installed, we install docker-compose using pip:

sudo pip install  docker-compose

Now, create a directory that will be used for all this stuff and a subdirectory for each of the components. I will call it /IOTServer, but you can choose any other name and parent directory:

<pre>sudo mkdir /IOTServer
sudo chown MYUSERID:MYUSERID /IOTServer
cd /IOTServer
mkdir grafana
mkdir influxdb
mkdir mqtt
mkdir node-red
mkdir portainer
mkdir telegraf<br>

Copy the configuration file docker-compose.yaml to /IOTServer. We are going to use it to download all the required images (this is one of the features of docker-compose):

docker-compose pull

This will last for a while, as all the container images are downloaded to the Raspberry. When pull process is finished, it is time to configure each component:

Step 6: Installing and Configuring Containers

Now, we will install and setup containers for each service
2.1 Influxdb

First, we have to create a configuration file. We do it by starting the container with 'config' option, we use -rm option in docker run to delete the container once it has finished.

docker run --rm influxdb influxd config > /IOTServer/influxdb/influxdb.conf

Now, start the influxdb for the first time to create a database ('admin' is a suggestion; pay attention to password and database name). When output stabilizes and database creation has finished, you may stop it with control-C.

docker run --rm -v /IOTServer/influxdb/influxdb.conf:/etc/influxdb/influxdb.conf -v /IOTServer/influxdb:/var/lib/influxdb -e INFLUXDB_DB=mydatabasename -e INFLUXDB_ADMIN_USER=admin -e INFLUXDB_ADMIN_PASSWORD=the_password influxdb -config /etc/influxdb/influxdb.conf /init-influxdb.sh

To start influxdb:

 docker-compose up -d influxdb

2.2 Mosquitto

Create a file named /IOTServer/mqtt/conf/mosquitto.conf with the following contents:

persistence true
persistence_location /mqtt/data/
 allow_anonymous true
#user mosquito
# Port to use for the default listener. 
port 1883 
log_dest stdout 
#listener 9001 
#protocol websockets

Mosquitto is now ready to start

2.3. NODE-RED

start Node-red container in background:

docker-compose up -d node-red Edit /IOTServer/node-red/settings.js

and look for (if you don't have rw access give it to the directory) and uncomment this section:

// Securing Node-RED    
// To password protect the Node-RED editor and admin API, the following     
// property can be used. See "http://nodered.org/docs/security.html" for details.     
adminAuth: {
        type: "credentials", 
        users: [{
             username: "user admin", 
            password: "a hash uninteligible",
             permissions: "*"
        }]
    }

To create a valid hash, do (put the password you want to use instead of 'your-password':

docker exec -it node-red node -e "console.log(require('bcryptjs').hashSync(process.argv[1], 8));" your-password

Copy and paste the result in password field of /IOTServer/node-redsettings.js and restart node-red docker restart node-red with a Web browser, in http://nodered.org/docs/security.html node-red should ask for userid and password. Once connected you have an empty area to work with. Remember that if you are going to insert data in influxd database, you will need to install the influxdb module: Go to manage palette, then go to install tab and there look for influxdb and install it.

2.3 PORTAINER

Portainer is a tool that helps you navigate, inspect …etc. through dockers with a web browser instead of using the command line. It should have been downloaded when you did docker-compose pull. Let’s start it for the first time by:

docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock \
                --restart always --name portainer portainer/portainer -H \
                 unix:///var/run/docker.sock

Now Portainer is started and listening on port 9000. To navigate it just open a web browser on http://nodered.org/docs/security.html First time, you will be asked for a userID and password. The ones you put are set for the future.

2.4 TELEGRAF

I use Telegraf to collect workload metrics and store them in an influxdb database. Find below a simplified (reduced) configuration file. Before using it, don't forget to make a copy of the original telegraf.conf and to substitute My_IP_Address with your own ip address in the file provided.

Congratulations! You have set up an IOT server ... but, its only addressable from inside your network. To access it from internet, it's necessary to do some more things.

2.5 Domain registering

As it is not probable that you have a static address in internet, you need to setup a Dynamic DNS. This is a server that publishes the name you register with the IP address that you provide and updates it as the address change (you also have to provide the updated address). To accomplish this feature, I use duckdns (https://www.duckdns.org): it is free and works fine.

Just connect to https://www.duckdns.org and register the domain name you want. Prior to registering the domain name, you have to log in. It is possible to do it with your Reddit, Github, Google ..etc. account. After you register your domain, you get a token (if you forget it, you can find it in http://nodered.org/docs/security.html ) that will be needed in next steps. To tell duckdns your IPaddress, we will use a simple script.

First, create the directory: cd /IOTServer mkdir duckdns cd duckdns and create the script publish_ip.sh with a single line:

echo url="https://www.duckdns.org/update?domains=MYDOMAIN&token=YOURTOKEN&ip=" | curl -k -o /IOTServer/duckdns/publish_ip.log -K -

make it executable and add it to crontab:

chmod 700 publish_ip.sh

crontab -e

add to the end (run every 5 minutes)

*/5 * * * * /IOTServer/duckdns/publish_ip.sh >/dev/null 2>&1

You are telling duckdns what is your IP address so it can find refer that address with the name you are providing. 2.6 PORT FORWARDING

You probably have your raspberry behind a router that translates (NATs) your internal IP address into a public one. To make your Raspberry addressable, what we are going to do is to allow traffic HTTP and HTTPS to the Raspberry and then reroute it to the corresponding service using nginx. Port forwarding in your router is usually a simple task, you only have to configure on it to route 80 to [My_IP_Address:80] and and 443 to [My_IP_Address:443].

To implement SSL, I use certbot and letsencrypt: First of all, stop all services:

docker-compose down

Then download and install certbot:

<pre>cd /IOTServer<br> wget  <a href="http://nodered.org/docs/security.html" rel="nofollow"> http://nodered.org/docs/security.html </a> 
chmod 755 certbot-auto
./certbot-auto certonly --standalone --preferred-challenges http-01 --email mymailname@mailserver -d MYDOMAIN.duckdns.org

I can also set a certificate to every service:

<pre>./certbot-auto certonly --standalone --preferred-challenges http-01 --email mymailname@mailserver -d nodered.MYDOMAIN.duckdns.org <br>./certbot-auto certonly --standalone --preferred-challenges http-01 --email mymailname@mailserver -d portainer.MYDOMAIN.duckdns.org 
./certbot-auto certonly --standalone --preferred-challenges http-01 --email mymailname@mailserver -d grafana.MYDOMAIN.duckdns.org

These certificates must be renewed monthly, so add a cron task:

 * *  1 * *  /IOTServer/renew_cert.sh >/dev/null 2>&1

renew_cert.sh is a simple script to stop nginx, renew the certificate and start nginx again. edit renew_cert.sh and add #!/bin/bash /usr/bin/docker stop nginx /IOTServer/certbot-auto renew /usr/bin/docker start nginx and then

 chmod 700 renew_cert.sh

certificates are stored in Your certificate and chain have been saved at:

/etc/letsencrypt/live/SERVICE.MYDOMAIN.duckdns.org/fullchain.pem

Your key file has been saved at:

/etc/letsencrypt/live/SERVICE.MYDOMAIN.duckdns.org/privkey.pem

2.7 NGINX

We use nginx as reverse proxy, It was installed when running docker-compose pull. To configure it: edit /IOTServer/nginx/nginx.conf and uncoment line 23:

server_names_hash_bucket_size 64;

Then copy the file default to /IOTServer/nginx/site-confs/default (don't forget to change My_IP_Address with your own address)

2.8 GRAFANA

Only need to create an empty configuration file touch /IOTServer/grafana/grafana.ini

To start everything:

docker-compose up -d

Now you can access grafana at http://nodered.org/docs/security.html first time it will set the admin user and password with the data you provide

2.9 Autostart docker-compose

Now, it is necessary to start everything when Raspberry reboots. To do it we need to configure a service and to set it up.

<pre>cd /etc/systemd/system <br>sudo nano docker-compose-opt.service

add:

<pre># /etc/systemd/system/docker-compose-opt.service <br>[Unit]
Description=Docker Compose Opt Service 
Requires=docker.service 
After=docker.service 
[Service] 
Type=oneshot
RemainAfterExit=yes 
WorkingDirectory=/IOTServer 
ExecStart=/usr/local/bin/docker-compose up -d 
ExecStop=/usr/local/bin/docker-compose down 
TimeoutStartSec=0
[Install] 
WantedBy=multi-user.target

And we enable it

sudo systemctl enable docker-compose-opt

To test cd /IOTServer docker-compose down sudo reboot now on reboot, we check if everything is running: docker ps or navigating to http://nodered.org/docs/security.html and navigating to http://nodered.org/docs/security.html or http://nodered.org/docs/security.html Now you have an IOTserver were to send data from your gauges, work with them and store if you want in an influxdb database.

docker-compose.yaml
Download

Step 7: Configuring Node-red

Now you have your IOT server up and running, it is time to receive, transform and store the data that the meteo gauge is sending to it. As said at the beginning, node-red is a very friendly environment to program; there are some blocks especially useful when dealing with these kinds of projects, e.g.

mqtt-in. Connects to a MQTT broker and subscribes to messages from the specified topic.
JSON. Converts between a JSON string and its JavaScript object representation, in either direction.
influxdb. A simple output node to write values and tags to an influxdb measurement.

I will go through each of them to configure the required values. To connect to your node-red server, open in a browser http://[My_IP_Address]:1880. It will ask for userID and password, put the ones you defined when configuring node-red container. You will be presented a blank canvas, here is where we will define our flow. Block mqtt in

We will begin with “mqtt in” block. This block will: 1. Connect to a mqtt broker and subscribe to the message topic that our client is using. To subscribe , drag mqtt in node to the canvas and double click on it. In “Topic” write the topic that the client is using (meteo/envia in this case), and a name for the node. This name is to represent the node so can be anyone. Regarding QoS, I recommend using ‘2’ that means that one and only one message will arrive .

Next step is to 2. Define a mqtt broker to connect to. In the previous window, click on the pencil in the combo “Server”. Probably it will say something like ”add a new mqtt-broker”. You will get another window, where we will define the broker to connect. In “name”, put any name that has some meaning, e.g. “Sensor”. In Server, you have to put the local address of the mqtt broker. In this case, it will be the local address of the Raspberry. Por must be 1883, unless you changed it when configuring mosquito (not likely and not recommended). Check “use clean session”. For entry configurations, we will keep “Client ID” blank, so the node will generate a random ID, leave SSL/TLS unchecked and do not configure Security nor Messages tabs. Once you are familiar with mqtt, it will be time to secure connections. After saving and deploying (there is a button in the upper right are of the main node-red window) if mosquito is running you will see a “connected” and a green square below mqtt node.

Block JSON

This is a very simple one. It will convert a JSON string into a JSON object. Drag a JSON node for “function” area to the canvas and click on it. In “action”, select “convert between JSON String &object”. The property to convert is msg.payload (the message the client is sending), keep unchecked “Format JSON String” checkbox.

Block influxdb

We will now configure a node to insert in a database the data sent that was received in a message stored in msg.payload and then formatted to a JSON object (and passed again in msg.payload within node-red). First, you will probably need to install influxdb node. To do it, go to “manage palette” (in dropdown menu in upper right side of the window) and there, in “install” tab, look for influxdb. Once found, install it.

Once installed, you can select t for the left menu. As with previous nodes, drag in into the canvas and double click on it. There is a server to set ( I will cover it soon), a measurement name (this is an influxdb terminology, it is somewhat like a table, where data is to be stored) and a node name, that is not relevant in configuration terms. Leave “Advanced Query options” unchecked and click on the pencil in “Server” (probably you will see there * Add new influxdb* ).

When clicked, you will get another window where to configure the database. In this window you must set the host, with the Raspberry IP address and the database name you defined when configuring influxdb.

Summary

Find below these three blocks configured in node-red. Meteo-client represents the connection with the MQTT Broker. To check how received messages are being formatted, I configured a debug node (‘formatted debug’) which output goes to the debug window.

[{"id":"54133c00.4e5ce4","type":"tab","label":"Flow
2","disabled":false,"info":""},{"id":"5364ed36.6c7294","type":"mqtt
in","z":"54133c00.4e5ce4","name":"meteo-Client","topic":"meteo/envia","qos":"1","broker":"d7c0ce0a.414658","x":170,"y":200,"wires":[["711d918b.3ee22"]]},{"id":"fed2a0b5.cd474","type":"debug","z":"54133c00.4e5ce4","name":"formated
debug","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","x":490,"y":320,"wires":[]},{"id":"d3ae7cf.1bb068","type":"influxdb
out","z":"54133c00.4e5ce4","influxdb":"10f94ce.9562bb3","name":"database","measurement":"meteo","precision":"","retentionPolicy":"","x":470,"y":380,"wires":[]},{"id":"711d918b.3ee22","type":"json","z":"54133c00.4e5ce4","name":"","property":"payload","action":"","pretty":false,"x":330,"y":260,"wires":[["fed2a0b5.cd474","d3ae7cf.1bb068"]]},{"id":"d7c0ce0a.414658","type":"mqtt-broker","z":"","name":"Sensor","broker":"192.168.1.11","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"1","birthPayload":"{}","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""},{"id":"10f94ce.9562bb3","type":"influxdb","z":"","hostname":"192.168.1.11","port":"8086","protocol":"http","database":"yourdatabase","name":"yourdatabase","usetls":false,"tls":""}]

You might need to change database name (and password) and also the
broker IP address.

Meteo-client represents the connection with the MQTT Broker. To check how messages are being formatted, I configured a debug node (‘formatted debug’) which output goes to the debug window.

Step 8: Configuring Grafana

As data is stored in influxdb database, it can be visualized with Grafana. Grafana is easy to customize so you can have personalized charts, there is plenty of documentation and shared charts. To have some starting point, my main charts for meteo gauges are the following ones (I have three gauges, only one of them has rain gauge):

The JSON object that represents this
dashboard is very long, should you want to download it, it is attached as meteo_home.json