Introduction: Setting Up an MQTT Broker. Part 2 : IoT, Home Automation
Okay In 'Part 1 : IoT, Home Automation', you've set up and configured your 'Vanilla' Raspberry Pi (pardon the pun) and can connect to it remotely with PuTTY. Now we will set up a server which will establish your main household IoT infrastructure and test it works as expected.
This Instructable documents how to set up an IoT MQTT Broker and verify it is working correctly by 'snooping' MQTT topics with MQTTSpy.
What Parts do I need?
- A home network
- Router with DCHP server
- One Raspberry Pi B+ or better
- One USB Hub, Keyboard, Mouse and LCD monitor to set up the Pi
- One Win 7 PC
What skills do I need?
- Top of the list :- Patience
- Some understanding of TCP/IP
- A very basic understanding of what client/server means
- Small knowledge of Linux
IoT devices rely upon an underlying protocol named 'Message Queuing Telemetry Transport' or 'MQTT' for short. This protocol is known as 'Publish/Subscribe' and is extremely simple and lightweight, being designed specifically for 'constrained devices' and low bandwidth systems. Which in normal language means a communications protocol so simple and small that even a microprocessor system like an ESP8266 or Arduino can support it. Once you couple this with a WiFi connection (ESP8266) or Ethernet link (Ethernet Shield) you have yourself the basis of an IoT device.
The second picture above is a generalised pictorial representation of a Publish/Subscribe protocol in action.
Here an IoT device or 'Client' has detected an unusually high temperature and 'Published' this 'Topic' to the MQTT 'Broker'. The MQTT Broker has then forwarded this information to all those 'Clients' who have chosen to 'Subscribe' to this news 'Topic'. In this case it could be an intelligent device which switches on an A/C unit or a passive data logging system or even an intelligent building management system which in its wisdom has chosen to open the windows.
Equally it could have been an engine management system publishing fuel consumption data or a smart meter publishing electricity consumption figures.
If this explanation is troubling you then think of it like YouTube. I upload a video to YouTube (the broker) and my many followers who have subscribed to my channel will get a notification of the new publication.
This is the heart of the publish/subscribe protocol. The MQTT Broker is optimally designed to handle many thousands of concurrently connected MQTT clients.
It is primarily responsible for receiving all messages, filtering them, deciding who is interested in them and forwarding messages to all subscribed clients.
The MQTT Client, be it Subscriber or Publisher (or both in one device) is any device from small Microcontroller up to a fully fledged server, that has an MQTT library running and is connected to an MQTT Broker over any kind of network.
And that's all you need to know on MQTT. We won't be going into the protocol or how they plug in as there is simply no need at this point.
To Part 1 : IoT - 'Intranet' of Things : Home Automation. Part 1 : IoT, Introduction
To Part 3 : Creating Your First IoT WiFi Device. Part 3 : IoT, Home Automation
Step 1: Installing Mosquitto MQTT Broker on Your Raspberry Pi
We will be using the Mosquitto MQTT Broker from Eclipse as its free and runs under Linux. It is possible to use a PC based MQTT Broker however I specifically wanted to use a Raspberry Pi as they are cheap, physically very small and pack a large punch where processing power is concerned, better still I have a load of them handy. I used a B+ Pi.
I am assuming you have followed the steps in my earlier Instructable named 'IoT - Intranet of Things : Home Automation. Part 1 : IoT, Introduction' and have a Raspberry Pi already set up and ready to go.
All you need to do is change the host name from 'MYHOSTNAME' to 'MQTTSVR'. You can do this by using PuTTY to connect to the Pi and from the command line enter;
- sudo raspi-config
Select (9) Advanced Options then (A2) Hostname.
Enter 'MQTTSVR' and save the changes. Exit then choose to reboot the Pi
You can now re-connect PuTTY to this Pi by using the new hostname 'MQTTSVR.local'. Create a PuTTY saved session named 'MQTTSvr', it will come in handy later.
The installation process is as follows;
Connect to your new MQTT server via PuTTY and remembering to copy from this Instructable and paste into PuTTY to avoid having to type any commands by hand.
Ensure you are in the home directory. If you are you will see the following prompt
If you don't see this enter
- cd ~
Now complete the following commands;
- sudo apt-key add mosquitto-repo.gpg.key
Now make this repository available to apt;
- cd /etc/apt/sources.list.d/
As we are using the Raspberry Pi Jessie Debian image enter the following;
Now enter the proceeding commands, which may take some time to complete. Also answer Yes (Y) to any questions;
- sudo apt-get update
- sudo apt-get upgrade
To complete the install process enter the following, answering Yes (Y) to any questions;
- sudo apt-get install mosquitto
Let the Pi do its thing and complete where you will see the command prompt.
That's it you're done. Mosquitto has now been installed as a service and will automatically start when you boot your Raspberry Pi.
Now we need to test it works.
Note : Before you remove power from your MQTT Broker enter the following command and wait for the Pi to shutdown.
- sudo shutdown now
If you want to quickly connect to your Raspberry Pi server using PuTTY without having to type in username and password each time. Do the following, assuming you have already created a saved session named 'MQTTSvr' as in Step 7, 'Part 1 : IoT, Automation' and your MQTTSVR username and password are unchanged from the example given.
Locate your PuTTY install and do the following;
- Create a short cut to it.
- Right click on the short cut
- Select the 'Shortcut' tab
- In the text box named 'Target:' Enter the following after the text "...\putty.exe"
- -load MQTTSvr -l pi -pw raspberry
- So in my case I installed to "C:\Program Files (x86)\PuTTY" the text will read;
- "C:\Program Files (x86)\PuTTY\putty.exe" -load MQTTSvr -l pi -pw raspberry
- Now drop this short cut on your desktop and double click each time you want to open a PuTTY session.
Step 2: MQTT Topics
This is a good time to have a quick discussion regarding MQTT Topics.
What is a Topic?
A topic is a string made up of UTF-8 (Unicode Transformation Format 8-Bit) characters and is the data sent by a Publisher to a Broker to be forwarded on to any Subscribers. In short its a basic text string.
How are they constructed?
Topics are broken down into one or more topic level each with a minimum of at least 1 character. These levels are separated by a forward slash '/' and can include ' '(spaces).For example (not including the quote marks);
'this/is/a/six/level/topic example with spaces'
Though arguably not very useful topic, hopefully gets the point over. Topics are case sensitive so;
is different to;
This is a single topic
as are these;
' ' (space) and '/'
There is no special action necessary to create a topic, the act of subscription or publication does this for you.
And that's about it.
Is there a special format or best practice for constructing topics?
The short answer is 'no' and there are lots of views surrounding how you should make them up (and I will cover the topic in a later Instructable). However you would need to consider the following when creating your topics;
Keep them short. They are sent over the network and need to be parsed (read) by some device/broker/server, so the longer and more intricate you make them the harder they are to decode.
Where you can, use meaningful names. It makes debugging easier. As a human its a lot simpler to type, or read 'sensor1' than 'Wcp10X!3wdrka2'.
Make them have meaning to you. The examples I have seen tend to be like a kind of address such as;
'myhouse/first floor/bedroom one/ceiling/light'
This assumes a fixed location IoT device and for the most part this will be true, though this type of nomenclature fails when considering mobile or re-deployable IoT devices, such as a temperature humidity sensor, or switchable extension socket. Should you decide to move a switchable extension socket from the living room to the bedroom to replace a failed device, you will need to reprogram the IoT device to respond to its new location to be consistent.
For my series on IoT based home automation I will be using topics which refer to the device only such as;
This is because I ultimately want to use a MySQL database to hold the configuration and IoT device information. Which I will hopefully cover later.
Oh, and one final thing, it is usually convention not to precede the first topic with a '/', but you can see I've ignored this. :-)
Are there any short cuts?
Yes, MQTT offers what is know as topic 'wild cards'. These are special characters a subscriber can place into a subscribing topic to reduce the need to subscribe to many items. There are two types of wildcard characters; '+' known as 'Single Level' and '#' for 'Multi Level'. The Picture above gives examples of both.
A word of caution. If you use the '#' multi level wild card and have a lot of IoT devices in your system, then be prepared for a lot of messages!
Are there any special topics?
Yes, topics beginning with '$' are reserved for internal statistics of the MQTT Broker. At the time of writing this Instructable there is no clear official standardisation of topics that must be published by a Broker. However it is common practice to prefix with '$SYS/' for this type of information. The mosquitto MQTT Broker conforms to this '$SYS/broker/#'. (see Linux mosquitto man page for full details. ie. man mosquitto)
So if you want to see all the mosquitto MQTT Broker statistics you would need to subscribe MQTTSpy to the following (see next steps for further details);
Step 3: Setting Up Your Test Environment
To test our new MQTT Broker we will first install a client which can subscribe and publish to a given topic on the Raspberry Pi. We will then monitor this exchange via tool named MQTTSpy from the PC (just like in the first picture above).
Why, test the install? Well it could save you a lot of time in the long run. To correctly set up a robust home automation system you start to rely upon and to which in the fullness of time be adding lots of useful IoT clients to, you need to make sure that all the links in the chain are working correctly, that way if you observe an issue with your new IoT device you can quickly rule out as many systems as possible and zero in on the actual issue.
Installing Pub/Sub Clients (Raspberry Pi)
From a PuTTY window enter the following command on your MQTT Broker. Ensure you are in the home directory;
if not enter;
- cd ~
- sudo apt-get install mosquitto-clients
Installing MQTTSpy (PC)
In order to successfully run MQTTSpy you will need to have the Java Runtime Engine (JRE) greater than version 8 installed on your PC.
To check what version of the JRE you have installed, run up a 'dos in a box', ie a command line window (picture above).
From the Start button, enter into the search box with the grey text 'Search programs and files'
and hit enter. A dos in a box will appear into which you will need to type;
It will return 'java version "W.X.Y.ZZ"' and some other 'guff' (picture above) or an error. if you have anything other that greater than version 8 (actually 1.8.y.zz) you will need to follow these instructions;
Go to this Oracle downloads web page.
Select the 'Accept Licence Agreement' radio button, download and install the 'Java SE Runtime Environment 8u92', jre-8u92-windows-x64.exe (as in the picture above).
Ok, now go to the MQTTSpy download page given below;
Select and download mqtt-spy jar file (picture above). At the time of writing version 0.5.0 was the latest available.
Save your 'mqtt-spy-0.5.0-jar-with-dependancies.jar' file to any directory.
To get the software to run you can double click it. It this doesn't work you can enter the following from the command line, in the directory you saved the jar file.
java -jar mqtt-spy-0.5.0-jar-with-dependancies.jar
Alternatively you could unzip and drop the RunMQTTSpy.bat file (given below) into the same directory and double click this to run MQTTSpy each time.
Ok, you 've now installed all your test software, let's configure it and run a test.
To open a command prompt at a specific directory do the following;
- Open windows explorer
- Navigate to the directory of interest
- Press and hold 'Alt' and 'Shift' keys
- Right click on the directory of interest
- From the dropdown menu select 'Open command window here'
Step 4: Configuring MQTTSpy and Testing Your MQTT Broker
Out of the box, the minimal configuration required by MQTTSpy is as follows;
- Connecting to your MQTT Broker
- Subscribe to a given topic
- Optionally, publish to a given topic
Here the video above shows how to connect to our mosquitto MQTT Broker 'MQTTSVR' and add a wild card to view all test topics and test the MQTT Broker. The running order is as follows;
- Open two auto login PuTTY sessions
- Start MQTTSpy by double clicking the 'RunMQTTSpy.bat' file (I've done the video as if you have just used MQTTSpy for the first time, so you can see the initial configuration)
- Create a standard configuration
- Create a new MQTT Broker connection which we will connect to our Broker
- Enter the Broker details 'MQTTSVR.local' and give it a client ID 'Win 71'
- Then connect to the Broker 'Open connection' and save this new set up so we can use it later.
- Select this Broker tab, which has gone 'GREEN' to show it has connected to MQTTSVR
- Make a subscription to all test topics (with multi level wild card)
- From a PuTTY session window make a subscription to the test topic (no wildcards).
- mosquitto_sub -t "/test/Topic"
- From the other PuTTY session window publish a message to the test topic.
- mosquitto_pub -t "/test/Topic" -m "Hello World"
- You will now see this Hello World message arrive at both our client subscribers (PuTTY and MQTTSpy)
- Now to prove we can publish a message from a client not located on the MQTT Broker we copy the test topic to the publish box in MQTTSpy and send the message 'SteveQuinn' to the MQTT Broker
- You can now see this message arrive both at the PuTTY subscribed client and the MQTTSpy client (here MQTTSpy has become both subscriber and publisher).
- Finally we close the connection to the MQTT Broker from the control panel and quit MQTTSpy.
- Followed by closing the two PuTTY sessions. Remember to use ^C (Ctrl C) to quit the subscription before quitting.
Ok, so now we are accomplished MQTT Broker testers. What does this give us?
If we are having any issues with our Home Automation set up, we can now quickly run up MQTTSpy and check to see if the Broker is healthy or any published messages are 'malformed' etc. We can even publish messages, meaning we can test any IoT device we create.
Step 5: There Are IOS MQTT Clients
Although in the examples I have given above I use PC based software to carryout my testing equally it is possible to use a smartphone app.
Pictured above is one such application named 'MQTTool' for the iOS. Although not as feature rich as MQTTSpy it is easy to use and allows you to do some basic testing.
Step 6: And Not to Leave Out Android Users
From the play store I could see quite a few free apps, I tried many out but found the app pictured above to be the best of them 'MQTT Dashboard'. Though you need to put in an IP Address not the host name 'MQTTSVR.local' it does have a better method to publish to topics, allowing you to create text boxes, switches, sliders etc.
I included a screen grab from the Pi detailing how to get the IP Address of the Pi. To do this, connect via putty and enter;
The Pi will respond with the screen shot above and I have highlighted the IP address you will need to enter into MQTT Dashboard. I have assumed you are connecting to your network via Ethernet. If you have connected via WiFi read 'wlan0' for 'eth0'
Step 7: So What's Next?
In the next three Instructables we will create three simple IoT devices for your Intranet which are based around the ESP8266-01 and the Arduino ATMega2560 with Ethernet Shield. The first of which we will revisit to improve the design and extend its use to show just how easy it is to make your own IoT devices.
Step 8: References Used
I used the following references to put this Instructable together;
Official MQTT site;
MQTTSpy Home Page;
MQTTSpy Downloads, Getting Started Guide and further reading;
Good place for MQTT Clients;
MQTT Spy advanced usage
Mosquitto Eclipse Home Page
Mosquitto MQTT Broker details, downloads, install process etc.;
Good background material
Good source of best practice for Topic naming conventions
Google groups protocol discussions
3 years ago
On my version of Raspbian the configuration tool doesn't seem to have an option to change the host name.
Advanced is #7 but there is no host name there.
sudo nano /etc/hosts and changed it on the last line.
I hope it doesn't kill it.
Oops. wrong place.
It was raspberrypi I changed to MQTTSVR
Undid the change in host but I don't remember if it should have been raspberry or raspberrypi
Reply 3 years ago
It's been a while since I wrote the article. For mDNS to work, you will need hostname separation. I use hostnames when debugging (MQTTSpy) and linking together various servers (OpenHAB, Mosquitto, Grafana and InfluxDB). It just makes life easier entering the name as opposed to an IP address (which can change after the DHCP lease expires). You shouldn't have multiple devices with the same host name (for obvious reasons). That said however, I have used fixed IP for my IoT devices as I couldn't get hostnaming to work with the PubSub Client on the Arduino (I keep meaning to look into this, but it's not high on my list of things to do). In effect I use the MAC address of my MQTT server to force my router to issue the same IP address to the MQTT server. It's a nasty fix but it works. If you are using the latest version of Rasbian (stretch), try option 2 Network Options, then N1 Hostname.
Reply 3 years ago
Thanks. It appears I have things mucked up.
I seem to have two IP addresses for my Pi.
$ hostname -I
I think I have a lot of unmucking to do.
5 years ago
MQTT works on the pi. I'm also using MQTTBox, aswell as mqtt-spy, it appears to be quicker to install on windows10.
6 years ago
i believe its "sudo apt-get install mosquitto" becuase yours only had 1 t and i beleve its wrong
Reply 6 years ago
You are correct, actually there are a few other 't'-'tt' errors, I believe it was caused by IE, I now use Chrome for all my edits. The Instructable has been updated to reflect this.
Thanks for the notification.