How to Use MQTT With the Raspberry Pi and ESP8266

31,970

121

25

Published

Introduction: How to Use MQTT With the Raspberry Pi and ESP8266

About: I'm a 19-year-old aspiring Physicist, who enjoys coding and making stuff!

In this Instructable, I will explain what the MQTT protocol is and how it is used to communicate between devices.Then, as a practical demonstration, I shall show you how to setup a simple two client system, where an ESP8266 module will send a message to a Python program when a button is pushed. Specifically, I am using an Adafruit HUZZAH module for this project, a Raspberry Pi and a desktop computer. The Raspberry Pi will be acting as the MQTT broker, and the Python client will be run from a separate desktop computer (optional, as this could be run on the Raspberry Pi).

To follow along with this Instructable, you will need to have some basic knowledge of electronics, and how to use the Arduino software. You should also be familiar with using a command line interface (for the Raspberry Pi). Hopefully, once you've gained the knowledge of what MQTT is, and how to use it in a basic scenario, you will be able to create your own IoT projects!

Required Parts

  • 1 x Raspberry Pi, connected to a local network (running Jessie)
  • 1 x ESP8266 Module (Adafruit HUZZAH)
  • 1 x Breadboard
  • 3 x Jumper Wires (Male-to-Male)
  • 1 x Pushbutton
  • 1 x 10k Ohm Resistor (Brown-Black-Orange colour code)

I've created this Instructable, as MQTT has always interested me as a protocol and there are many different ways it could be used. However, I couldn't seem to get my head around how to code devices to use it. This was because I didn't know/understand what was actually going on to take my "Hello, World!" from device A and send it to device B. Hence, I decided to write this Instructable to (hopefully) teach you how it works, and to also reinforce my own understanding of it!

Step 1: What Is MQTT?

MQTT, or MQ Telemetry Transport, is a messaging protocol which allows multiple devices to talk to each other. Currently, it is a popular protocol for the Internet of Things, although it has been used for other purposes - for example, Facebook Messenger. Interestingly MQTT was invented in 1999 - meaning it's as old as me!

MQTT is based around the idea that devices can publish or subscribe to topics. So, for example. If Device #1 has recorded the temperature from one of its sensors, it can publish a message which contains the temperature value it recorded, to a topic (e.g. "Temperature"). This message is sent to an MQTT Broker, which you can think of as a switch/router on a local area network. Once the MQTT Broker has received the message, it will send it to any devices (in this case, Device #2) which are subscribed to the same topic.

In this project, we will be publishing to a topic using an ESP8266, and creating a Python script that will subscribe to this same topic, via a Raspberry Pi which will act as the MQTT Broker. The great thing about MQTT is that it is lightweight, so it perfect for running on small microcontrollers such as an ESP8266, but it is also widely available - so we can run it on a Python script as well.

Hopefully, at the end of this project, you will have an understanding of what MQTT is and how to use it for your own projects in the future.

Step 2: Installing the MQTT Broker on the Raspberry Pi

To setup our MQTT system, we need a broker, as explained in the previous step. For the Raspberry Pi, we will be using the "Mosquitto" MQTT broker. Before we install this, it is always best to update our Raspberry Pi.

sudo apt-get update
sudo apt-get upgrade

Once you've done this, install mosquitto and then the mosquitto-clients packages.

sudo apt-get install mosquitto -y
sudo apt-get install mosquitto-clients -y

When you've finished installing these two packages, we are going to need to configure the broker. The mosquitto broker's configuration file is located at /etc/mosquitto/mosquitto.conf, so open this with your favourite text editor. If you don't have a favourite text editor or don't know how to use any of the command line editors, I'll be using nano so you can follow along:

sudo nano /etc/mosquitto/mosquitto.conf

At the bottom of this file, you should see the line:

include_dir /etc/mosquitto/conf.d

Delete this line. Add the following lines to the bottom of the file.

allow_anonymous false
password_file /etc/mosquitto/pwfile
listener 1883

By typing those lines, we've told mosquitto that we don't want anyone connecting to our broker who doesn't supply a valid username and password (we'll get on to set these in a second) and that we want mosquitto to listen for messages on port number 1883.

If you don't want the broker to require a username and password, don't include the first two lines that we added (i.e. allow_anonymous... and password_file...). If you have done this, then skip to rebooting the Raspberry Pi.

Now close (and save) that file. If you are following along with the nano example, press CTRL+X, and type Y when prompted.

Because we've just told mosquitto that users trying to use the MQTT broker need to be authenticated, we now need to tell mosquitto what the username and password are! So, type the following command - replacing username with the username that you would like - then enter the password you would like when prompted (Note: if, when editing the configuration file, you specified a different password_file path, replace the path below with the one you used).

sudo mosquitto_passwd -c /etc/mosquitto/pwfile username

As we've just changed the mosquitto configuration file, we should reboot the Raspberry Pi.

sudo reboot

Once the Raspberry Pi has finished rebooting, you should have a fully functioning MQTT broker! Next, we are going to try to interact with it, using a number of different devices/methods!

Step 3: Testing the Broker

Once you've installed mosquitto on the Raspberry Pi, you can give it a quick test - just to make sure everything is working correctly. For this purpose, there are two commands that we can use on the command line. mosquitto_pub and mosquitto_sub. In this step, I will guide you through using each of these to test our broker.

In order to test the broker, you will need to open two command line windows. If you are using Putty or another SSH client, this is as simple as opening another SSH window and logging in as usual. If you are accessing your Pi from a UNIX terminal, this is exactly the same. If you are using the Raspberry Pi directly, you will need to open two terminal windows in the GUI mode (the command startxcan be used to start the GUI).

Now that you have opened two windows, we can get started on the testing. In one of the two terminals, type the following command, replacing username and password with the ones you setup in the previous step.

mosquitto_sub -d -u username -P password -t test


If you decided not to set a username and password in the previous step, then from now on, ignore the -u and -P flags in the commands. So, as an example, the mosquitto_sub command would now be:

mosquitto_sub -d -t test


The mosquitto_sub command will subscribe to a topic, and display any messages that are sent to the specified topic in the terminal window. Here, -d means debug mode, so all messages and activity will be output on the screen. -u and -P should be self-explanatory. Finally, -t is the name of the topic we want to subscribe to - in this case, "test".

Next, in the other terminal window, we are going to try and publish a message to the "test" topic. Type the following, remembering again to change username and password:

mosquitto_pub -d -u username -P password -t test -m "Hello, World!"


When you press enter, you should see your message "Hello, World!" appear in the first terminal window we used (to subscribe). If this is the case, you're all set to start working on the ESP8266!

Step 4: Setting Up the ESP8266 (Adafruit HUZZAH)

This step if specific to the Adafruit HUZZAH (as that is what I am using to complete this project). If you are using a different Arduino / ESP8266 device, you may wish to skip this step. However, I would advise you skim read it, just in case there is any information here that may be relevant to you.

For this project, I am going to be programming the HUZZAH with the Arduino software. So, if you haven't already, make sure to install the Arduino software (newer than 1.6.4). You can download it here.

Once you have installed the Arduino software, open it and navigate to File->Preferences. Here you should see (near the bottom of the window) a text box with the label: "Additional Boards Manager URLs". In this text box, copy and paste the following link:

http://arduino.esp8266.com/stable/package_esp8266com_index.json


Click OK to save your changes. Now open the Board Manager (Tools->Board->Board Manager) and search for ESP8266. Install the esp8266 by ESP8266 Community package. Restart the Arduino software.

Now, before we can program the board, we need to select a few different options. In the Tools menu option, select Adafruit HUZZAH ESP8266 for Board, 80 MHz for the CPU Frequency (you can use 160 MHz if you wish to overclock it, but for now I'm going to use 80 MHz), 4M (3M SPIFFS) for the Flash Size, and 115200 for the Upload Speed. Also, make sure to select the COM port that you are using (this will depend on your setup).

Before you can upload any code, you need to make sure that the HUZZAH is in bootloader mode. To enable this, hold down the button on the board marked GPIO0, and whilst this is held, hold down the Reset button as well. Then, release the Reset button, and then GPIO0. If you have done this correctly, the red LED that came on when you pressed GPIO0 should now be dimly lit.

To upload code to the microcontroller, first make sure the HUZZAH is in bootloader mode, then simply click the upload button in the Arduino IDE.

If you are having any trouble setting up the HUZZAH, further information can be found at Adafruit's own tutorial.

Step 5: Programming the ESP8266

Now we will begin to program the ESP8266, but before we can start, you will need to install the following libraries in the Arduino Library manager (Sketch->Include Libraries->Manage Libraries)

  • Bounce2
  • PubSubClient


Once you've installed those libraries, you will be able to run the code I've included in this Instructable (MQTT_Publish.zip). I've made sure to comment it so that you can understand what each section is doing, and this should hopefully enable you to adapt it to your needs.

Remember to change the constants at the top of the code so that your ESP8266 can connect to your WiFi network and your MQTT Broker (the Raspberry Pi).

If you decided not to set a username and password for the MQTT Broker, then download the MQTT_PublishNoPassword.zip file instead.

Step 6: Installing Python Client (paho-mqtt)

Thankfully, this step is very simple! To install the mosquitto python client, you just need to type the following into the command line (Linux/Mac) or even command prompt (Windows).

pip install paho-mqtt


Note: Windows command prompt may have an issue running the pip command if you didn't specify that you wanted pip installed and python added to your PATH variable when you installed Python. There are a number of ways of fixing this, but I think just reinstalling Python is the easiest way. If in doubt - give it a google!

Step 7: Python Client - Subscribing

In this step, we are going to setup the Python script (either on the Raspberry Pi itself or on another computer connected to the network) to handle all of the messages that are sent (published) by the ESP8266 to the MQTT topic.

I have included the python code below (PythonMQTT_Subscribe.py), which has been commented to help you understand what is going on, but I will explain some of the main features here as well.

If you didn't set a username and password for the MQTT connection earlier, download the PythonMQTT_SubscribeNoPassword.py file instead.

Step 8: Communicating Between ESP8266 Devices

If you want to set up an IoT network, for example, you may wish to communicate between ESP8266 devices. Thankfully, this isn't much more complex than the code we've written before, however, there are a couple of notable changes.

For one ESP to send data to another, the first ESP will need to publish to the topic, and the second ESP will need to subscribe to that topic. This setup will allow for a one-way conversation - ESP(1) to ESP(2). If we want ESP(2) to talk back to ESP(1), we can create a new topic, to which ESP(2) will publish, and ESP(1) will subscribe. Thankfully, we can have multiple subscribers on the same topic, so if you want to send data to a number of systems, you will only need one topic (to which they all subscribe, except the device which is sending the data, as that will be publishing).

If you need help figuring out what each device needs to do, think about the system as a room of people. If ESP(1) is publishing, you can imagine this device as a "speaker", and any devices that are subscribing to the topic are "listeners" in this example.

I have included some example code below, which demonstrates how an ESP8266 can subscribe to a topic, and listen for certain messages - 1 and 0. If 1 is received, the on-board LED (for the HUZZAH - GPIO 0) is switched on. If 0 is received, this LED is switched off.

If you want to process more complex data, this should be done in the ReceivedMessage function (see code).

For your own projects, if you need to both send and receive data, you can incorporate the publish function from the previous example into the code included in this step. This should be handled in the main Arduino loop() function.

Remember to change the variables at the top of the code to suit your network!

2 People Made This Project!

Recommendations

  • Clocks Contest

    Clocks Contest
  • Woodworking Contest

    Woodworking Contest
  • Oil Contest

    Oil Contest
user

We have a be nice policy.
Please be positive and constructive.

Tips

Questions

25 Comments

hi! As a newbie to ESP8266 I have a question. Can I run the publisher and subscriber code on the same ESP? since my ESP is a publisher and a subscriber at the same time & I am facing a lot of confusion on how to do it.

7 replies

Apologies for my horrendously late response to your question!

I don't have an ESP8266 to hand at the moment to test it out, but try copying the Publisher loop() code and putting it just before client.loop() is called in the Subscriber code for the ESP8266. You'll also have to copy over the button variables and the bouncer code for it to work in the Subscriber code.

That way, I think the code should (1) check if it's connected to WiFi, (2) check if the button is pressed and handle the publishing side, and finally (3) check for new messages using the client.loop() function before repeating all that over again (back to (1)).

Hope that helps you (and works)! :)

Thanks! It worked perfectly! I have another question please. Since in this tutorial we have learnt to connect a button & use it to publish messages, can i also connect a coin vibrator on, say, GPIO12? so it vibrates each time I send a message to esp using python publisher code. would that be possible? would connecting a vibrator require some libraries like you did for the button?

Eagerly waiting for your response.

Glad it worked for you! As long as the voltage and current requirements for the coin vibrator don't exceed the limits of the ESP8266's pins (I'm sure they won't but it's best to check), that should all be fine, and you shouldn't need to use any additional libraries. In the MQTT_Subscribe code for the ESP8266, you'll need to switch GPIO12 on and off rather than the ledPin (either set up a new variable like we did with the ledPin or replace digitalWrite(ledPin, ...); with the number 12). If you wanted to make it vibrate every time a message was sent, then just do the vibration outside of the if statement (digitalWrite a HIGH, then do a delay(n) then digitalWrite LOW to buzz the coin vibrator for n milliseconds), and that way it will trigger no matter the message content. Also, make sure to setup GPIO12 as an output though in void setup(), like has been done with the ledPin. Good luck with your project! :)

Thanks a lot! I have tested it on my esp & it works great! So when this whole mechanism worked fine with a single esp, I connected another one, say esp2 & subscribed it to the same topic. so whenever I publish a message to topic, both of them can vibrate. The problem that I am facing is, esp1 receives messages & vibrates but esp2 doesn't. Although it is connected to wifi, has a unique IP assigned & all but it still doesn't respond to any message.

can you please help me to figure this out? I am trying really hard to find a solution but can't find out what the problem might be.

Hmm... That is odd! In the ESP8266 code, do both devices have different clientID values? (near the top of the code) e.g. One of the ESP8266s has a clientIDof "ESP8266_1" and the other has "ESP8266_2" (change the value before uploading the code to each one). That's the only thing I can think of that might be causing a problem. At least the coin vibrator works for you! :)

Thank you so much! It worked! I couldn't believe I ignored this part. Thanks again!

You're welcome! I'm pleased that's fixed it for you. :)

thanks for the instructable!! ;-)

Could someone explain how I can send an integer from esp to raspberry?

I always get the following error:
exit status 1
invalid conversion from 'int' to 'const char*' [-fpermissive]

I'd like to send temperature data

Thanks for your help!

1 reply

Hi. Glad you enjoyed the Instructable! Take a look at this solution I found on the Arduino forums. Essentially, you want to put your message (in your case, a temperature) into a String, and then use the built-in toCharArray() method on your string object to convert the temperature data into a form you can send. :)(https://forum.arduino.cc/index.php?topic=241222.msg1729997#msg1729997)

Hi,

I first proposed to use the Moquette broker due to the fact it is lightweight.

In the same use I think to use Crouton Dashboard to get/use the MQTT values (as actors for some). I did not yet test it in place of using a Home Automation box (Domoticz, Jeedom,Home Automation, OpenHAB, ...) because they use a huge part of memory and harddrive space.

This is why I though to use the Crouton Dashboard :

http://crouton.mybluemix.net/

If some of you may confirm it works to act with this tutorial it should be appreciate to get a good approach ;-)

I ask this because I thing trying to use on the same RPi 3 :

- a lightweight broker (perhaps Moquette or uTT)
- the needed Dashboard to be able to read/act data values (Crouton)
- to allow Kalliope to read MQTT (and/or act) values : https://kalliope-project.github.io/

Have a nice day.

I've never heard of those before, but thanks for letting me know about them! Yeah, they should work perfectly fine (as far as I know), and you should be able to keep the ESP8266 code the same. The only difference is you'll have to change the MQTT broker setup, but that'll be explained on their websites. :)

i am not under stand how to update data of esp8266 to file which is in raspberry pi3

and iam finding error in Python code

please help me to solve this

File "/home/pi/Desktop/PythonMQTT_Subscribe.py", line 22

print "Connected!", str(rc)

^

SyntaxError: invalid syntax

1 reply

I'm so sorry for taking so long to respond to you!

You might be using a different Python version (Python 3), in which case you'll need to change all of the print statements in the Python code. For example, here you'd need to change the line to read:

print("Connected!", str(rc))

Try adding brackets to all of the print statements like in that example.

Hope that helps! :)

hey! Is there a possibility that both of the codes could work simultaneously or could merge? since my ESP is a subscriber and a publisher at the same time for different topics.

So is that possible? Please respond.

Hi! Thanks a lot for sharing this article. I learned a lot about MQTT. Can you please tell me, It is possible that 2 ESP8266 (both act as clients) communicate with each other through MQTT broker? Means there will not any App or web command, Everything will be written on publisher and subscriber firmware. Ex: ESP (01) will send the message LED_ON with topic and ESP (02) will be subscribed to the topic and will receive the message and then the led on GPIO02 of ESP(02) will lit up.

Please tell me. Sorry If I'm asking you something silly. I'm in learning phase of MQTT protocol so not completely getting the idea about MQTT.

3 replies

Hi! That's not a silly question at all. In fact, it has made me realise that I've left out quite a key topic! I'll update the Instructable to (hopefully) answer your question. :)

Wow! superb. Thanks a ton for sharing this :)

No worries. Glad to have helped! :)