Introduction: Raspberry Pi NRF24L01+ Data Collector Using Google Forms
A headless Raspberry Pi with an nRF24L01+ 2.4GHz radio transceiver, connected to the internet. Receiving wireless data packets from remote sensors and presenting the data to a Google form for subsequent viewing and presentation.
The data is posted to the Google Form using a Python script, there is also an nCurses dashboard for the Raspberry Pi data collector that may be accessed through a VNC server, or SSH.
Step 1: Parts List
The parts list assumes familiarity with the chosen sensor.
- Raspberry Pi (any model)
- nRF24L01+ (enough for the Raspberry Pi and the Arduino sensors deployed)
- ATmega328 based sensor (Arduino any model)
- nRF24L01+ ATmega328P-PU radio sensor, 1 , Instructable , $12.69 each
- Raspberry Pi nRF24L01+ Mini-Hat/Proto-Board , 1 , Instructable , $7.44 each
Some Sensors
- DHT11
- photo resistor
- DS18B20
Also required
- Google account
- Internet service
I am using the Raspberry Pi nRF24L01+ mini project board I designed in my previous Instructable, along with my ATmega328P-PU radio sensor from another Instructable.
Step 2: Set Up a Google Form
Assuming that we already have a Google account, it is time to create a Form to collect sensor data. The following assumes some familiarity with Google Form creation.
Set up a form to collect the sensor data:
From the Google Drive, or from the Google Apps menu select:
even more from Google,
Forms
This will bring you to a new:
Untitled Form
Now add the following Form Short Answer Questions:
sensor identity
DHT11 Temperature
DHT11 Humidity
photo resistor
Raspberry Pi Internal Temperature
Raspberry Pi DHT11 Temperature
Raspberry Pi DHT11 Humidity
reported by
Now obtain the link to the form:
from the menu select
Get pre-filled link
Fill in the responses with meaningful descriptions, something that will help us determine what the entry should be. This will help us later in determining the url addresses for the Python script.
Click on
Submit
copy the link url address
https://docs.google.com/forms/d/1onnROUjj1Orjoge0STvMRA7PDDR-YKw5eoo2mruLSCQ/viewform?entry.1183120755=Sid&entry.408289807=DHT11-TC&entry.2133582784=DHT11_H%25&entry.181535051=DS18B20C&entry.1756888165=PHOTO&entry.1187558071=startTime-millis&entry.498543388=nowTime-millis&entry.1127599175=RPiDHT11C&entry.751073501=RPiDHT11H%25&entry.1895537677=RPi_C&entry.773327888=RPI-submitter
Now we shall break the url down to the component parts. We will need these later when we type the Python script.
Sheet Identity: 1onnROUjj1Orjoje0STvMRA7PDDR-YKw5eoo2mruLSCQ
and the form entry components
Sensor ID:&entry.1183120755=Sid
Sensor DHT11 Temperature:&entry.408289807=DHT11-TC
Sensor DHT11 Humidity:&entry.2133582784=DHT11_H%25
Sensor DS18B20 Temperature:&entry.181535051=DS18B20C
Sensor Photo Resistor:&entry.1756888165=PHOTO
Sensor startTime:&entry.1187558071=startTime-millis
Sensor nowTime:&entry.498543388=nowTime-millis
RPi DHT11 Temperature:&entry.1127599175=RPiDHT11C
RPi DHT11 Humidity:&entry.751073501=RPiDHT11H%25
RPi Internal Temperature:&entry.1895537677=RPi C
RPI Submitter:&entry.773327888=RPi-submitter
We now have the component address pointers for our Python script, which we will create in step 6.
Step 3: Link a Sheet to the Form Responses
From the form edit menu
select:
Responses tab.
select the green sheet icon:
View responses in sheets
Now we have a spread sheet of our Form Responses.
Step 4: Create Sheets to Manipulate and Display the Data
Now we will create a sheet to display the individual sensor data from the main Parent Data Collection Form.
on the form right click on the Responses tab
right click on the green Sheet icon, this will create a Google spread sheet displaying the response data.
Once we have this sheet displayed, click in the address bar of the web browser and copy the sheet url address.
https://docs.google.com/spreadsheets/d/1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XU/edit#gid=1234567890
We need this in order to create a sheet to import the responses into. This will allow us to display the responses as charts and tables.
Now we may break the url down to the component parts. We will need these later when we type the Python script.
- Sheet Identity: 1xXB_AaugW4lHesegjUV2PkormimJgg1VqTutIz9y5XU
In order to initially link a NEW SHEET to the parent sheet, place this importrange command in a cell. I placed this command in cell A5.
=importrange(ʺ1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XUʺ,ʺForm Responses 1!A:Lʺ)
initially the cell will show
#REF
right click in the cell
You need to connect these sheets
Allow access
right click on Allow access
Now the sheets are linked, this sheet will update and the Parent Sheet data will be imported, but we would like to select the individual sensors and the data associated with them. So we will edit the cell with importrange and create a query.
Now edit the cell
=query(importrange(ʺ1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XUʺ,ʺForm Responses 1!A:Lʺ),ʺSelect* Where Col2 contains 'ʺ&B2&ʺ'ʺ,1)
This will allow us to enter the sensor number in cell B2, the sheet will then pull data from the Parent Sheet and create the assigned charts.
=query(importrange(ʺ1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XUʺ,ʺForm Responses 1!A:Lʺ),ʺSelect* Where Col2 contains 'ʺ&B2&ʺ'ʺ,1)
Now create some charts, the pictures above give some examples. Familiarity with Microsoft Excel or Open Office should allow this quite intuitively.
Step 5: Libraries: Raspberry Pi
We now need to install some libraries to our Raspberry Pi Base Station. If you already have the Optimized nRF24L01 and nCurses libraries installed you will most probably want to skip this section.
Either from a remote computer on the network:
ssh pi@192.168.0.101 (your Raspberry Pi network address)
Password: raspberry (or whatever you have set the password to be)
or from the Raspberry Pi itself
sudo apt-get update sudo apt-get upgrade
sudo apt-get install python-pip
sudo apt-get install python-dev sudo apt-get install python-setuptools
sudo apt-get install libboost1.50-all
Install the Optimized nRF24L01 Libraries
cd ~ wget https://tmrh20.github.io/RF24Installer/RPi/install.sh chmod +x install .sh
./install.sh
answer the questions as prompted with either a Y or N, to install the required libraries. At a minimum the RF24 libraries should be installed.
rm install.sh cd rf24libs cd RF24
sudo make
sudo make install
cd examples_RPi
sudo make
sudo make install
Now install the python RF24 Wrapper
cd ~ cd rf24libs/RF24/RPi/pyRF24
sudo ./setup.py build sudo ./setup.py install
cd ~
Now check to see that all of the RF24 libraries have been configured correctly.
cd rf24libs/RF24/examples_RPi
sudo ./pingpair_dyn.py
We should have the nRF24 summary displayed, see the example picture above.
Note: if you have used the connections for the nRF24L01+ I have described here, then the example program will give a HARDWARE error if you try to operate it. I am using different CE, CSN connections. However the summary does indicate that the libraries are all installed correctly, as in the picture above. See my instructable for the modification to pingpair_dyn.py, this will then function with the connections I have used.
Install the curses Libraries
nCurses may be installed from the Optimized nRF24L01+ installation script, we should answer Y to the prompt for installation of the Library.
Install the future Library
sudo pip install future
Install the Adafruit_DHT Library
cd ~ git clone https://github.com/adafruit/Adafruit_Python_DHT.git sudo apt-get update sudo python setup.py install
Install the requests Library
sudo pip install requests
You may find that some of these libraries are already installed on your distribution, if so, then the system will notify you.
We should now have all of our libraries installed.
Step 6: Raspberry Pi : Base Receiver
The Raspberry Pi Base Receiver uses nCurses to display the data dashboard. Python code completes the data read and fills in the Google form.
Import the libraries
from __future__ import print_function import RPi.GPIO as GPIO import time import os import Adafruit_DHT import curses import requests import time from RF24 import *
Post the data to the form. The form entries are defined in the submissions string. The url address of the form is obtained from the pre-filled submission menu option. This address may then be broken down to obtain the individual form items.
https://docs.google.com/forms/d/1onnROUjj1Orjoge0STvMRA7PDDR-YKw5eoo2mruLSCQ/viewform?entry.1183120755=Sid&entry.408289807=DHT11-TC&entry.2133582784=DHT11_H%25&entry.181535051=DS18B20C&entry.1756888165=PHOTO&entry.1187558071=startTime-millis&entry.498543388=nowTime-millis&entry.1127599175=RPiDHT11C&entry.751073501=RPiDHT11H%25&entry.1895537677=RPi_C&entry.773327888=RPI-submitter
The individual form items may then be coded as follows, where data represents the data variable to be posted to the form
'entry.1234567890': data
The data is then posted to the form using the library function
requests.post(formResponseurl, submissions)
def postData(): submissions = {'entry.1183120755': Sid , 'entry.408289807': temperatureC , 'entry.2133582784': humidity , 'entry.181535051': ds18b20C , 'entry.1756888165': Apin0 , 'entry.1187558071': startTime , 'entry.498543388': thisTime , 'entry.1127599175': TempC , 'entry.751073501': HUM , 'entry.1895537677': RPiC , 'entry.773327888': Submitter} response = requests.post(formResponseurl, submissions) return 1
This is the main loop, reading the data from the sensors, posting to the Google form and displaying on the nCurses dashboard.
while True: pipe = [0] while not radio.available(): time.sleep(2) nCursesSleeping() length = radio.getDynamicPayloadSize() buffer = radio.read(length) nCursesRadio() flashLEDs(3) Sid = buffer[0] + 256*buffer[1] startTime = buffer[2] + 256*buffer[3] thisTime = buffer[4] + 256*buffer[5] Apin0 = buffer[6] + 256*buffer[7] Apin1 = buffer[8] + 256*buffer[9] Apin2 = buffer[10] + 256*buffer[11] Apin3 = buffer[12] + 256*buffer[13] Apin4 = buffer[14] + 256*buffer[15] Apin5 = buffer[16] + 256*buffer[17] Apin6 = buffer[18] + 256*buffer[19] Apin7 = buffer[20] + 256*buffer[21] temperatureC = buffer[22] + 256*buffer[23] humidity = buffer[24] + 256*buffer[25] ds18b20C = buffer[26] + 256*buffer[27] # now read the DHT11 sensor on the Raspberry Pi readDHT11(pDHT11) readPi = readPiTemp() RPiC = float(readPi) RPiF = (RPiC * 1.8) + 32.0 nCursesWindow() mscreen.refresh() mscreen.clear() postIt = postData() nCursesWindow()
Note there is a warning presented after the data has been posted to the Form. This may be ignored, however should this type of security warning be of concern then an alternative method for data posting should be employed. For example this Instructable.
Attachments
Step 7: Libraries: ATmega328P-PU (Arduino)
We will assume that we are using Arduino IDE 1.6.5.
Optimized nRF24L01 Libraries
- Download the Optimized nRF24L01 libraries from here.
- Unzip the file
- Rename the folder to RF24.
- Copy this folder to the library folder of the Arduino working folder.
Adafruit_DHT Libraries
Start the Arduino IDE
select the menu items
Sketch
Include Library
Manage Libraries
From the Library Manager type DHT into the search filter
- Install the Adafruit DHT Unified Library 1.0.0
- Install the DHT Sensor Library 1.1.0
- Restart the Arduino IDE
One Wire DS18B20 Libraries
From the Library Manager type one wire into the search filter
- Install the One Wire Library 2.3.0
- Restart the Arduino IDE
AVR Standard Libraries
From the Library Manager type avr into the search filter
- Install the AVR Standard C Time Library 1.8.1
- Restart the Arduino IDE
Step 8: Make Component Connections
Connect the nRF24L01+ to the Raspberry Pi
- nRF24L01+ pin 1, GND connect to Pin 6, GND on the Raspberry Pi
- nRF24L01+ pin 2, 3V3 connect to pin 1, 3V3 on the Raspberry Pi
- nRF24L01+ pin 3, CE connect to pin22, GPIO25 on the Raspberry Pi
- nRF24L01+ pin 4, CSN connect to pin 24, GPIO8 (CE0) on the Raspberry Pi
- nRF24L01+ pin 5, SCK connect to pin 23, GPIO11 (SCLK) on the Raspberry Pi
- nRF24L01+ pin 6, MOSI connect to pin 19, GPIO10 (MOSI) on the Raspberry Pi
- nRF24L01+ pin 7, MISO connect to pin 21, GPIO9 (MISO) on the Raspberry Pi
- nRf24L01+ pin 8, IRQ, this pin is not connected to the Raspberry Pi
Connect LEDs to the Raspberry Pi
- LED1 connect the anode to pin 33, GPIO13 on the Raspberry Pi
- LED2 connect the anode to pin 29, GPIO5 on the Raspberry Pi
Connect the DHT11 to the Raspberry Pi
- DHT11 data pin connect to pin 7, GPIO4 on the Raspberry Pi
Connect the nRF24L01+ to the ATmega328/Arduino
- nRF24L01+ pin 1, GND connect to GND on the Arduino
- nRF24L01+ pin 2, 3V3 connect to 3V3 on the Arduino (or a 3V3 source)
- nRF24L01+ pin 3, CE connect to pin13, D7 on the Arduino
- nRF24L01+ pin 4, CSN connect to pin 14, D8 on the Arduino
- nRF24L01+ pin 5, SCK connect to pin 19, D13 on the Arduino
- nRF24L01+ pin 6, MOSI connect to pin 17, D11 on the Arduino
- nRF24L01+ pin 7, MISO connect to pin 18, D12 on the Arduino
- nRf24L01+ pin 8, IRQ, this pin is not connected to the Arduino
Connect some sensors to the ATmega328/Arduino
- DHT11 pin 2, DATA connect to pin 4, D2 on the Arduino
- photo resistor, connect to pin 23, A0 on the Arduino
- DS18B20 DATA pin, connect to pin 6, D4 on the Arduino
Connect some sensors to the Arduino, the schematic above illustrates the common sensor arrangement I am using. Reference to my previous Instructable or to other more detailed instructables will help. Consider that the sketch should be modified to incorporate other sensors as required.
Step 9: ATmega328P-PU: Remote Sensor Transmitter
Transmit the sensor data to the Raspberry Pi base receiver.
Note: if we do not have any sensors, the basic routine will work. Provided that the nRF24L01+ has been connected. All of the analog pins are read and sent to the Raspberry Pi. Analog 0 is defined as the photo resistor, so that will be posted to the Google Form. The other sensors will register a zero if nothing is attached to the pins.
Libraries
#include <SPI.h> #include <avr/sleep.h> #include "nRF24L01.h" #include "RF24.h" #include <printf.h> // Printf is used for debug #include <Adafruit_Sensor.h> #include <DHT.h> #include <DHT_U.h> #include <OneWire.h>
Read and prepare the data, most of the sensor routines read data into global variables that will be passed to the radio payload. There is a lot of room for improvement with the programming here.
Obtain the Temperature and Humidity from the DHT11 sensor and temperature from the DS18B20 sensor.
getTemperatureHumidity(); getDS18B20();
Read all of the analog pins
int Apin0 = analogRead(A0); // A0 is the photo resistor (light sensor) int Apin1 = analogRead(A1); int Apin2 = analogRead(A2); int Apin3 = analogRead(A3); int Apin4 = analogRead(A4); int Apin5 = analogRead(A5); int Apin6 = analogRead(A6); int Apin7 = analogRead(A7); thisTime = millis();
Create the payload as an array of integers, each item is two bytes long. Consisting of a Low byte and a High byte. The final integer result may be obtained from:
Low byte + 256* High byte
int send_payload[] = {Sensor, startTime, thisTime, Apin0, Apin1, Apin2, Apin3, Apin4, Apin5, Apin6, Apin7, temperatureC, humidity, DS18B20C};
The payload size for this generic data sensor payload is 28 bytes
int payload_size = sizeof send_payload; radio.stopListening();
then send it
radio.write( send_payload , payload_size );
Download the sketch file and copy it to the Arduino working directory. Load the sketch into the Arduino IDE and then flash it to the Arduino sensor.
Step 10: Collect the Data and Feed the Form
Download the Python script RPi_logger5.py from Step 6. Rename the file to something more readable.
For example rpiforms.py.
Then from the Raspberry Pi navigate to the Python script downloaded.
Open a terminal then:
sudo python rpiforms.py
Start some sensors and watch the Raspberry Pi pull the data from the collected feeds and enter them into their respective google form fields. We may now view the data as it is collected from our Google Docs or Google Drive.
Here are the shared Links to the Form and the data sheet:
The Data Collection Form [Link].
The Data Viewing sheet [Link]
Step 11: Observations and Conclusions
Data visualization
We are collecting the data on one pipe and use Google sheets to split out each sensor from the Parent Form data and view it graphically. This allows quite rapid inference. For example the data collected here illustrates that the time intervals are perhaps a little too rapid. On the charts the data appears to have been collected at the same time interval, but has generated different temperature and humidity readings. In reality the readings are seconds apart,the chart scale covering the whole time frame of the data is giving this impression. However Google sheets allows manipulation of the Chart scales dependent on the display desired.
This data may be viewed and interrogated from almost anywhere. All that is required is access to the internet and Google Sheets.
Goals Met
The data is readily available from Google sheets
- Charts may be readily created
- Trends and patterns are easily observed
- Data may be readily shared
- The data posting method is relatively easy and does not require AUTH0 manipulation
Applications
- Laboratory Temperature, Humidity, Light, Bath Monitoring
- Home Automation, room monitoring
- Weather Station
Improvements
- Improve and clean the Python script
- Expand the nCurses Python script to scan more pipes
- Expand the nCurses Python script to create a dashboard menu allowing control of sensor or groups of sensors
- Introduce robust error handling
- Allow dynamic data handling
Improve and clean the Arduino sensor sketch
- Fix a standard sensor set and introduce more robust sensor routines to cover them
- Remove and clean up the Serial.print statements, a lot of them are there for debug
- The power saving routines are not implemented well, improve it, this will involve rethinking some of the sensors and hardware.
- The sensor identification is assigned at compile time, this is not the best way to perform this function, keep the sensor sketch generic, but allow the sensors to be assigned to/from eeprom.
- Overall polish for general deployment
Overall I have been able to monitor several sensors in real time using a very simple spread sheet. This has served my purpose extremely well.