Introduction: IOT123 - ASSIMILATE SENSOR HUB: ICOS10 CUSTOMIZATION WEBSEREVER

By bend. break, blend
About: The tension between novelty and familiarity...

The ASSIMILATE SENSOR/ACTOR Slaves embed metadata that is used for the defining visualizations in Crouton. This build adds a webserver to the ESP8266 Master, serves some config files that can be modified by the user, then uses those files to redefine the visualizations. So the names of the dashboard cards and most of the configurable properties can be changed. This was necessary e.g. the DHT11 publishes Temperature and Humidity properties: if a site has several nodes with separate DHT11 sensors they all can't be called Temperature (Garage Temp., Yard Temp...). The metadata length restriction set by the I2C Bus (16 chars) does not exist and richer values can be applied (up to 64 chars).

Optional Basic Authentication is cofigurable for the edit webpage, as well as an exclusion list from Authentication for other resources.

A low-side switch that powers down the slaves when necessary, has also been developed on an existing daughter-board.

As a technical note, before starting this build the memory footprint was 70% because of a global metadata object graph. The latest AssimilateBus library has had breaking changes that decouple the global variable into smaller JSON files saved to SPIFFS. This has brought the footprint back to ~50%, which is safer for all of the JSON parsing/building. The AssimilateBusSlave library remains the same (ASSIM_VERSION 2) throughout these changes.


FEATURES AND VISION

Currently the Slaves (sensors and actors) are self contained and rely on convention based I2C messages to read properties or act on commands. The Master picks up the metadata and properties from slaves and sends it to a MQTT broker. It also starts a webserver and serves JSON files that can be edited to configure the master and customize the metadata/properties that eventually is consumed by Crouton. The individual sensors/actors are read/commanded via Crouton without the master having any prior knowledge of what the slaves do.

One of the goals of the ASSIMILATE IOT NETWORK is to customize Crouton so that mashup editors served from the IOT NODE webservers (like this build), are added as webcomponents that will give complete control of what the thing does i.e. the master is not programmed, the slaves have basic feature sets but the Crouton dashboard embeds all the business rules needed to run the thing!

The Crouton fork is seen as an option for decentralized control/configuration of things. In essence any MQTT client/GUI combination can administer your things, as every function (sensors and actors) are exposed as MQTT endpoints.

Step 1: Crouton

Crouton. http://crouton.mybluemix.net/ Crouton is a dashboard that lets you visualize and control your IOT devices with minimal setup. Essentially, it is the easiest dashboard to setup for any IOT hardware enthusiast using only MQTT and JSON.

The ASSIMILATE SLAVES (sensors and actors) have embedded metadata and properties that the master uses to build up the deviceInfo json packet that Crouton uses to build the dashboard. The intermediary between ASSIMILATE NODES and Crouton is a MQTT broker that is websockets friendly: Mosquito is used for the demo.

As the ASSIMILATE MASTER (this build) requests properties, it formats the response values in the required format for Crouton updates.

Step 2: The Webserver / Editor

As the Master boots (this build) an embedded webserver is started. The IP Address is output to the Serial Console; eventually this will be published to the Crouton dashboard.

When you browse to the URL stated, the ACE EDITOR will be loaded:

Ace is an embeddable code editor written in JavaScript. It matches the features and performance of native editors such as Sublime, Vim and TextMate.

Ace is popular with embedded webservers and gives a good interface for editing and saving the JSON files.

Clicking on a filename on the left will read the file from SPIFFS on the ESP8266, and load the content for editing on the right. The file can be saved from the top toolbar.

To upload a file:

  1. Choose File from your local file system.
  2. Insert a folder path (if needed) in the textbox.
  3. Click Upload.
  4. Refresh the page.

Step 3: Device Customization

The configuration of the device (the ESP8266) is actioned through the device.json file.

Some of these entries (wifi_ssid, wifi_key) will need to be modified before you upload the data to SPIFFS (ESP8266 Sketch Data Upload).

Browse to the root of the webserver (shown in the console output like http://192.168.8.104/).


EDITING

In the ACE EDITOR choose config/device.json.

The entries are:

  • www_auth_username: authorization username for webserver files (empty for not authorization).
  • www_auth_password: authorization password for webserver files (if username defined).
  • www_auth_exclude_files: semi-colon delimited list of files paths to exclude from authorization checks (if username defined).
  • sensor_interval: the milliseconds between publishes of data to the MQTT Broker.
  • ntp_server_name: the name of the time server to use.
  • time_zone: the offset in hours for your local time.
  • wifi_ssid: the SSID of your local Access Point.
  • wifi_key: the key to use for the SSID.
  • mqtt_broker: the MQTT broker address.
  • mqtt_username: the username to use for the MQTT broker (empty for no account needed).
  • mqtt_password: the password to use from the MQTT username.
  • mqtt_port: the MQTT broker port.
  • mqtt_device_name: the name to use for MQTT topics and Crouton identification.
  • mqtt_device_description: the de3scription of the device shown in Crouton.
  • viz_color: the color to identify the device cards in Crouton (in the forked version)

Step 4: Property Customization

Each of the Slaves has a nvc struct array defined in the definitions.h file:

//--------------------------------------- PUBLISHED PROPERTIES

nvc props[2] ={
    {"Humidity", "", true},
    {"Temperature", "", false}
};
//--------------------------------------- END PUBLISHED PROPERTIES

Each of the entries has an index with the first one being 0 (zero).

The property name customization is actioned through the user_props.json file.

Browse to the root of the webserver (shown in the console output like http://192.168.8.104/).


EDITING

In the ACE EDITOR choose config/user_props.json (or upload one).

The structure is:

  • The first key is the Address of the Slave in the definitions.h file as identified by:
#define ADDRESS_SLAVE XX
  • The next level of keys is the Index of the Property.
  • The value of that key is the Property Name to use in Crouton instead of the Property Name defined in the definitions file.

Step 5: Metadata Customization

Due to the amount of possible customizations, each slave has its own metadata modification file. The files are to be in the format user_metas_.json.

The slave address is found in the definitions.h file in the ATTINY85 sketches:

#define ADDRESS_SLAVE XX

The metadata is defined in the same file like this:

const static char viz1[] PROGMEM = "VIZ_CARD_TYPE";
const static char viz2[] PROGMEM = "2:chart-donut";
const static char viz3[] PROGMEM = "1";

The first line is the name of the metadata item.

The second line is the value. It generally has a property index suffix.

The third line is the continuation flag. 1 - continue, 0 - end of metadata (VCC_MV).

Browse to the root of the webserver (shown in the console output like http://192.168.8.104/).


EDITING

In the ACE EDITOR choose config/user_metas_SLAVE_ADDRESS.json (or upload one). The structure is:

  • An array of name/value pairs.
  • Name is name of the metadata item to change.
  • Value is the change. The index suffix is checked for the replacement.

Step 6: Materials and Tools

ICOS10 (IDC) Shell Bill of Materials

  1. D1M BLOCK Pin Jig (1)
  2. D1M BLOCK base and housing (1)
  3. Wemos D1 Mini (1)
  4. Wemos D1 Mini Protoboard Shield (1)
  5. 40P Female Headers (8P, 8P, 9P, 9P)
  6. Male Header 90º (3P, 3P, 3P, 2P, 1P, 2P)
  7. 1" Double sided protoboard (2)
  8. 2N7000 NFET (1)
  9. 6 Pin Shrouded IDC Male Header (1)
  10. Hookup wire (~10)
  11. 0.5mm Tinned Wire (~4)
  12. 4G x 15mm Button head self tapping screws (2)
  13. 4G x 6mm self tapping countersunk screws (~20)
  14. Solder and Iron (1)

Step 7: MCU Preparation

In this build we are using the Wemos D1 Mini. If you have previously built a D1M WIFI BLOCK, you can use that for the modular hardware component. If not, as a bare minimum follow the next section.


SOLDERING THE HEADER PINS ON THE MCU (using the PIN JIG)
If you cant print a PIN JIG just follow the instructions and improvise: the height (offset) of the PIN JIG is 6.5mm.

  1. Print/obtain a PIN JIG from this page.
  2. Feed the header pins through bottom of the board (TX right-left) and into the solder jig.
  3. Press the pins down onto a hard flat surface.
  4. Press the board down firmly onto the jig.
  5. Solder the 4 corner pins.
  6. Reheat and re position board/pins if needed (board or pins not aligned or plumb).
  7. Solder the rest of the pins.


UPLOADING THE FIRMWARE

The code repository can be found here (snapshot).

A ZIP of the library can be found here (snapshot).

Instructions for "Importing a ZIP Library" here.

Once the library is installed you can open the example "mqtt_crouton_esp8266_customization_webserver".

Instructions for setting up Arduino for the Wemos D1 Mini here.

Dependencies: ArduinoJson, TimeLib, PubSubClient, NeoTimer (see attachments if breaking changes in repositories).


UPLOAD TO SPIFFS

Once the code has been loaded into the Arduino IDE, open device.json in the data/config folder:

  1. Modify the value of wifi_ssid with your WiFi SSID.
  2. Modify the value of wifi_key with your WiFi Key.
  3. Modify the value of mqtt_device_name with your preferred Device Identification (no joining needed).
  4. Modify the value of mqtt_device_description with your preferred Device Description (in Crouton).
  5. Save device.json.
  6. Upload the data files to SPIFFS.

Attachments

Step 8: MCU Housing Preparation

The MCU Housing exposes headers for the D1 Mini to plug into and headers for daughter-boards that communicate with the Socket (sensors and actors) circuit.



HOUSING HEADERS
This is based on a D1 Mini Protoboard, and breaks out these pins:

  1. Pins for the D1M WIFI BLOCK/D1 Mini to connect to.
  2. Direct Breakouts of the 2 rows of contacts from the D1M WIFI BLOCK/D1 Mini. These are only available for convenience whilst prototyping. It is expected the daughter-boards will block all access to these headers.
  3. 4 Breakouts of the specific pins used by the daughter-boards.


To add the D1M Contacts to the HOUSING HEADER:

  1. Watch the SOLDER USING THE SOCKET JIG video.
  2. Feed the header pins through bottom of the board (TX top-left on the top side).
  3. Feed jig over plastic header and level both surfaces.
  4. Turn jig and assembly over and firmly press header onto a hard flat surface.
  5. Press the board down firmly onto the jig.
  6. Solder the 4 corner pins using minimal solder (just temporary alignment of pins).
  7. Reheat and re position board/pins if needed (board or pins not aligned or plumb).
  8. Solder the rest of the pins.
  9. Remove the jig.
  10. Cut pins off above solders.


To add the Daughter-board Breakouts:

  1. Cut 4 off 9P Female headers.
  2. On the top, insert the 9P Headers as shown, and solder off on bottom.


To add the Direct Breakouts:

  1. Cut 2 off 8P Female headers.
  2. On the top, insert the 8P Headers as shown, and solder off on bottom.


To connect the headers, on the bottom with the TX pin oriented up:

  1. Trace and solder from the RST pin across 4 pins.
  2. Trace and solder from the A0 pin across 4 pins.
  3. Trace and solder from the D1 pin across 4 pins.
  4. Trace and solder from the D2 pin across 4 pins.
  5. Trace and solder from the D0 pin down 2 rows and across 4 pins.
  6. Trace and solder from the D7 pin across 4 pins.
  7. Trace and solder from the GND pin across 4 pins.
  8. Trace and solder from the 5V pin across 4 pins.
  9. Trace and solder from the 3V3 pin down 45° across 4 pins.



ASSEMBLING THE FIXTURE

The HOUSING HEADERS is affixed to the MCU HOUSING and this is affixed to the BASE PLATE.

  1. With the long side of the HOUSING HEADERS pointed to the hole, insert the D1M CONTACTS into the openings in the MCU HOUSING and push down flush.
  2. Insert the MCU onto the MCU CONTACTS during affixing to ensure the correct alignment.
  3. Place the HEADER FRAME over the top of the assemble fixtures and affix with 2 of 4G x 16mm screws.
  4. Place the assembled fixtures with the hole pointed toward the short side and affix with the 4G x 6mm screws.

Step 9: Building the Slaves Low-side Switch/RESET Daughter-board

This is an enhancement of the REST daughter-board developed last build. This adds a low-side switch that connects the slaves to GROUND. If the master is reset, the slaves will also, and the initialization propagating the metadata will start again.


ASSEMBLY

  1. On the inside, insert the 9P 90° Male Headers (1), 1P 90° Male Header (2), the 2N7000 (3), and solder off on the outside.
  2. On the inside, trace a yellow wire from YELLOW1 to YELLOW2 and solder.
  3. On the inside, trace a bare wire from SILVER1 to SILVER2 and solder.
  4. On the inside, trace a bare wire from SILVER3 to SILVER4 and solder.

Step 10: Assembling the Major Components

  1. Ensure the SHELL has been built and the circuit tested (cable and sockets).
  2. Swap out the 2P Male header on the 3V3 I2C DAUGHTER-BOARD with a 2P 90º 1P Male Header.
  3. Insert the 3V3 I2C DAUGHTER-BOARD, with the 3V3 pin on the ragged end of the headers (see pic).
  4. Insert the LOW-SIDE SWITCH/RESET DAUGHTER-BOARD, with the wire to the inside (see pic).
  5. Trace a Dupont lead between the 90º 1P Male Header on the RESET DAUGHTER-BOARD to the 3V3 I2C DAUGHTER-BOARD.
  6. Insert the IDC Socket from the SHELL CABLE into the IDC Header on the 3V3 I2C DAUGHTER-BOARD.
  7. Carefully insert the DAUGHTER-BOARDS/HOUSING between the cables in the SHELL and align the base holes.
  8. Fasten the BASE ASSEMBLY to the SHELL with the 4G x 6mm screws.
  9. Attach any ASSIMILATE SENSORS you have made.

Step 11: Next Steps

  1. Point your browser at http://crouton.mybluemix.net/crouton/connections.
  2. Ensure the broker is test.mosquitto.org.
  3. Click Connect.
  4. Enter the Device name input as mqtt_device_name in the /config/device.json file.
  5. Click Add Device.
  6. Click Auto-connect.
  7. Power up your ICOS10 (5V MicroUSB).
  8. Verify via the Crouton dashboard.

Check out these related builds.