Introduction: Raspberry Pi NRF24L01+ Data Collector Using Google Forms

A headless Raspberry Pi with an nRF24L01+ 2.4GHz radio transceiver, connected to the internet. Receiving wireless data packets from remote sensors and presenting the data to a Google form for subsequent viewing and presentation.

The data is posted to the Google Form using a Python script, there is also an nCurses dashboard for the Raspberry Pi data collector that may be accessed through a VNC server, or SSH.

Step 1: Parts List

The parts list assumes familiarity with the chosen sensor.

  1. Raspberry Pi (any model)
  2. nRF24L01+ (enough for the Raspberry Pi and the Arduino sensors deployed)
  3. ATmega328 based sensor (Arduino any model)
  • nRF24L01+ ATmega328P-PU radio sensor, 1 , Instructable , $12.69 each
  • Raspberry Pi nRF24L01+ Mini-Hat/Proto-Board , 1 , Instructable , $7.44 each

Some Sensors

  1. DHT11
  2. photo resistor
  3. DS18B20

Also required

  1. Google account
  2. Internet service

I am using the Raspberry Pi nRF24L01+ mini project board I designed in my previous Instructable, along with my ATmega328P-PU radio sensor from another Instructable.

Step 2: Set Up a Google Form

Assuming that we already have a Google account, it is time to create a Form to collect sensor data. The following assumes some familiarity with Google Form creation.

Set up a form to collect the sensor data:

From the Google Drive, or from the Google Apps menu select:

even more from Google,
Forms

This will bring you to a new:

Untitled Form

Now add the following Form Short Answer Questions:

sensor identity
DHT11 Temperature
DHT11 Humidity
photo resistor
Raspberry Pi Internal Temperature
Raspberry Pi DHT11 Temperature
Raspberry Pi DHT11 Humidity
reported by

Now obtain the link to the form:

from the menu select

Get pre-filled link

Fill in the responses with meaningful descriptions, something that will help us determine what the entry should be. This will help us later in determining the url addresses for the Python script.

Click on

Submit

copy the link url address

https://docs.google.com/forms/d/1onnROUjj1Orjoge0STvMRA7PDDR-YKw5eoo2mruLSCQ/viewform?entry.1183120755=Sid&entry.408289807=DHT11-TC&entry.2133582784=DHT11_H%25&entry.181535051=DS18B20C&entry.1756888165=PHOTO&entry.1187558071=startTime-millis&entry.498543388=nowTime-millis&entry.1127599175=RPiDHT11C&entry.751073501=RPiDHT11H%25&entry.1895537677=RPi_C&entry.773327888=RPI-submitter

Now we shall break the url down to the component parts. We will need these later when we type the Python script.

Sheet Identity: 1onnROUjj1Orjoje0STvMRA7PDDR-YKw5eoo2mruLSCQ

and the form entry components

Sensor ID:&entry.1183120755=Sid
Sensor DHT11 Temperature:&entry.408289807=DHT11-TC
Sensor DHT11 Humidity:&entry.2133582784=DHT11_H%25
Sensor DS18B20 Temperature:&entry.181535051=DS18B20C
Sensor Photo Resistor:&entry.1756888165=PHOTO
Sensor startTime:&entry.1187558071=startTime-millis
Sensor nowTime:&entry.498543388=nowTime-millis
RPi DHT11 Temperature:&entry.1127599175=RPiDHT11C
RPi DHT11 Humidity:&entry.751073501=RPiDHT11H%25
RPi Internal Temperature:&entry.1895537677=RPi C
RPI Submitter:&entry.773327888=RPi-submitter

We now have the component address pointers for our Python script, which we will create in step 6.

Step 3: Link a Sheet to the Form Responses

From the form edit menu

select:

Responses tab.

select the green sheet icon:

View responses in sheets

Now we have a spread sheet of our Form Responses.

Step 4: Create Sheets to Manipulate and Display the Data

Now we will create a sheet to display the individual sensor data from the main Parent Data Collection Form.

on the form right click on the Responses tab

right click on the green Sheet icon, this will create a Google spread sheet displaying the response data.

Once we have this sheet displayed, click in the address bar of the web browser and copy the sheet url address.

https://docs.google.com/spreadsheets/d/1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XU/edit#gid=1234567890

We need this in order to create a sheet to import the responses into. This will allow us to display the responses as charts and tables.

Now we may break the url down to the component parts. We will need these later when we type the Python script.

  • Sheet Identity: 1xXB_AaugW4lHesegjUV2PkormimJgg1VqTutIz9y5XU

In order to initially link a NEW SHEET to the parent sheet, place this importrange command in a cell. I placed this command in cell A5.

=importrange(ʺ1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XUʺ,ʺForm Responses 1!A:Lʺ)

initially the cell will show

#REF

right click in the cell

You need to connect these sheets
Allow access

right click on Allow access

Now the sheets are linked, this sheet will update and the Parent Sheet data will be imported, but we would like to select the individual sensors and the data associated with them. So we will edit the cell with importrange and create a query.

Now edit the cell

=query(importrange(ʺ1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XUʺ,ʺForm Responses 1!A:Lʺ),ʺSelect* Where Col2 contains 'ʺ&B2&ʺ'ʺ,1)

This will allow us to enter the sensor number in cell B2, the sheet will then pull data from the Parent Sheet and create the assigned charts.

=query(importrange(ʺ1xXB_AauqW4lHesegjUV2PkormimJgg1VqTutIz9y5XUʺ,ʺForm Responses 1!A:Lʺ),ʺSelect* Where Col2 contains 'ʺ&B2&ʺ'ʺ,1) 

Now create some charts, the pictures above give some examples. Familiarity with Microsoft Excel or Open Office should allow this quite intuitively.

Step 5: Libraries: Raspberry Pi

We now need to install some libraries to our Raspberry Pi Base Station. If you already have the Optimized nRF24L01 and nCurses libraries installed you will most probably want to skip this section.

Either from a remote computer on the network:

ssh pi@192.168.0.101 (your Raspberry Pi network address)
Password: raspberry (or whatever you have set the password to be)

or from the Raspberry Pi itself

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python-pip
sudo apt-get install python-dev
sudo apt-get install python-setuptools
sudo apt-get install libboost1.50-all

Install the Optimized nRF24L01 Libraries

cd ~
wget  https://tmrh20.github.io/RF24Installer/RPi/install.sh
chmod +x install .sh
./install.sh

answer the questions as prompted with either a Y or N, to install the required libraries. At a minimum the RF24 libraries should be installed.

rm install.sh
cd rf24libs
cd RF24
sudo make
sudo make install
cd examples_RPi
sudo make
sudo make install

Now install the python RF24 Wrapper

cd ~
cd rf24libs/RF24/RPi/pyRF24
sudo ./setup.py build
sudo ./setup.py install
cd ~

Now check to see that all of the RF24 libraries have been configured correctly.

cd rf24libs/RF24/examples_RPi
sudo ./pingpair_dyn.py

We should have the nRF24 summary displayed, see the example picture above.

Note: if you have used the connections for the nRF24L01+ I have described here, then the example program will give a HARDWARE error if you try to operate it. I am using different CE, CSN connections. However the summary does indicate that the libraries are all installed correctly, as in the picture above. See my instructable for the modification to pingpair_dyn.py, this will then function with the connections I have used.

Install the curses Libraries

nCurses may be installed from the Optimized nRF24L01+ installation script, we should answer Y to the prompt for installation of the Library.

Install the future Library

sudo pip install future

Install the Adafruit_DHT Library

cd ~
git clone  https://github.com/adafruit/Adafruit_Python_DHT.git
sudo apt-get update 
sudo python setup.py install

Install the requests Library

sudo pip install requests
You may find that some of these libraries are already installed on your distribution, if so, then the system will notify you.

We should now have all of our libraries installed.

Step 6: Raspberry Pi : Base Receiver

The Raspberry Pi Base Receiver uses nCurses to display the data dashboard. Python code completes the data read and fills in the Google form.

Import the libraries

from __future__ import print_function  
import RPi.GPIO as GPIO  
import time  
import os  
import Adafruit_DHT
import curses
import requests
import time from RF24 import * 

Post the data to the form. The form entries are defined in the submissions string. The url address of the form is obtained from the pre-filled submission menu option. This address may then be broken down to obtain the individual form items.

https://docs.google.com/forms/d/1onnROUjj1Orjoge0STvMRA7PDDR-YKw5eoo2mruLSCQ/viewform?entry.1183120755=Sid&entry.408289807=DHT11-TC&entry.2133582784=DHT11_H%25&entry.181535051=DS18B20C&entry.1756888165=PHOTO&entry.1187558071=startTime-millis&entry.498543388=nowTime-millis&entry.1127599175=RPiDHT11C&entry.751073501=RPiDHT11H%25&entry.1895537677=RPi_C&entry.773327888=RPI-submitter

The individual form items may then be coded as follows, where data represents the data variable to be posted to the form

'entry.1234567890': data

The data is then posted to the form using the library function

requests.post(formResponseurl, submissions)
def postData():
        submissions = {'entry.1183120755': Sid , 'entry.408289807': temperatureC , 'entry.2133582784': humidity , 'entry.181535051': ds18b20C , 'entry.1756888165': Apin0 , 'entry.1187558071': startTime , 'entry.498543388': thisTime , 'entry.1127599175': TempC , 'entry.751073501': HUM , 'entry.1895537677': RPiC , 'entry.773327888': Submitter}
	response = requests.post(formResponseurl, submissions)
	return 1

This is the main loop, reading the data from the sensors, posting to the Google form and displaying on the nCurses dashboard.

while True:     
    pipe = [0]  
    while not radio.available():  
        time.sleep(2)  
        nCursesSleeping()
        length = radio.getDynamicPayloadSize()  
    buffer = radio.read(length)
    
    nCursesRadio()
      
    flashLEDs(3)  
      
    Sid = buffer[0] + 256*buffer[1]  
    startTime = buffer[2] + 256*buffer[3]  
    thisTime = buffer[4] + 256*buffer[5]  
    Apin0 = buffer[6] + 256*buffer[7]  
    Apin1 = buffer[8] + 256*buffer[9]  
    Apin2 = buffer[10] + 256*buffer[11]  
    Apin3 = buffer[12] + 256*buffer[13]  
    Apin4 = buffer[14] + 256*buffer[15]  
    Apin5 = buffer[16] + 256*buffer[17]  
    Apin6 = buffer[18] + 256*buffer[19]  
    Apin7 = buffer[20] + 256*buffer[21]  
    temperatureC = buffer[22] + 256*buffer[23]  
    humidity = buffer[24] + 256*buffer[25]
    ds18b20C = buffer[26] + 256*buffer[27]  
      
    # now read the DHT11 sensor on the Raspberry Pi  
    readDHT11(pDHT11)  
    readPi = readPiTemp()  
    RPiC = float(readPi)  
    RPiF = (RPiC * 1.8) + 32.0  
    
    nCursesWindow()    
    mscreen.refresh()
    mscreen.clear()
    postIt = postData()    
    nCursesWindow()

Note there is a warning presented after the data has been posted to the Form. This may be ignored, however should this type of security warning be of concern then an alternative method for data posting should be employed. For example this Instructable.

Step 7: Libraries: ATmega328P-PU (Arduino)

We will assume that we are using Arduino IDE 1.6.5.

Optimized nRF24L01 Libraries

  1. Download the Optimized nRF24L01 libraries from here.
  2. Unzip the file
  3. Rename the folder to RF24.
  4. Copy this folder to the library folder of the Arduino working folder.

Adafruit_DHT Libraries

Start the Arduino IDE

select the menu items

Sketch
Include Library
Manage Libraries

From the Library Manager type DHT into the search filter

  1. Install the Adafruit DHT Unified Library 1.0.0
  2. Install the DHT Sensor Library 1.1.0
  3. Restart the Arduino IDE

One Wire DS18B20 Libraries

From the Library Manager type one wire into the search filter

  1. Install the One Wire Library 2.3.0
  2. Restart the Arduino IDE

AVR Standard Libraries

From the Library Manager type avr into the search filter

  1. Install the AVR Standard C Time Library 1.8.1
  2. Restart the Arduino IDE

Step 8: Make Component Connections

Connect the nRF24L01+ to the Raspberry Pi

  1. nRF24L01+ pin 1, GND connect to Pin 6, GND on the Raspberry Pi
  2. nRF24L01+ pin 2, 3V3 connect to pin 1, 3V3 on the Raspberry Pi
  3. nRF24L01+ pin 3, CE connect to pin22, GPIO25 on the Raspberry Pi
  4. nRF24L01+ pin 4, CSN connect to pin 24, GPIO8 (CE0) on the Raspberry Pi
  5. nRF24L01+ pin 5, SCK connect to pin 23, GPIO11 (SCLK) on the Raspberry Pi
  6. nRF24L01+ pin 6, MOSI connect to pin 19, GPIO10 (MOSI) on the Raspberry Pi
  7. nRF24L01+ pin 7, MISO connect to pin 21, GPIO9 (MISO) on the Raspberry Pi
  8. nRf24L01+ pin 8, IRQ, this pin is not connected to the Raspberry Pi

Connect LEDs to the Raspberry Pi

  1. LED1 connect the anode to pin 33, GPIO13 on the Raspberry Pi
  2. LED2 connect the anode to pin 29, GPIO5 on the Raspberry Pi

Connect the DHT11 to the Raspberry Pi

  1. DHT11 data pin connect to pin 7, GPIO4 on the Raspberry Pi

Connect the nRF24L01+ to the ATmega328/Arduino

  1. nRF24L01+ pin 1, GND connect to GND on the Arduino
  2. nRF24L01+ pin 2, 3V3 connect to 3V3 on the Arduino (or a 3V3 source)
  3. nRF24L01+ pin 3, CE connect to pin13, D7 on the Arduino
  4. nRF24L01+ pin 4, CSN connect to pin 14, D8 on the Arduino
  5. nRF24L01+ pin 5, SCK connect to pin 19, D13 on the Arduino
  6. nRF24L01+ pin 6, MOSI connect to pin 17, D11 on the Arduino
  7. nRF24L01+ pin 7, MISO connect to pin 18, D12 on the Arduino
  8. nRf24L01+ pin 8, IRQ, this pin is not connected to the Arduino

Connect some sensors to the ATmega328/Arduino

  1. DHT11 pin 2, DATA connect to pin 4, D2 on the Arduino
  2. photo resistor, connect to pin 23, A0 on the Arduino
  3. DS18B20 DATA pin, connect to pin 6, D4 on the Arduino

Connect some sensors to the Arduino, the schematic above illustrates the common sensor arrangement I am using. Reference to my previous Instructable or to other more detailed instructables will help. Consider that the sketch should be modified to incorporate other sensors as required.

Step 9: ATmega328P-PU: Remote Sensor Transmitter

Transmit the sensor data to the Raspberry Pi base receiver.

Note: if we do not have any sensors, the basic routine will work. Provided that the nRF24L01+ has been connected. All of the analog pins are read and sent to the Raspberry Pi. Analog 0 is defined as the photo resistor, so that will be posted to the Google Form. The other sensors will register a zero if nothing is attached to the pins.

Libraries

#include <SPI.h>
#include <avr/sleep.h> 
#include "nRF24L01.h"
#include "RF24.h"
#include <printf.h>  // Printf is used for debug 
#include <Adafruit_Sensor.h>
#include <DHT.h>
#include <DHT_U.h>
#include <OneWire.h>

Read and prepare the data, most of the sensor routines read data into global variables that will be passed to the radio payload. There is a lot of room for improvement with the programming here.

Obtain the Temperature and Humidity from the DHT11 sensor and temperature from the DS18B20 sensor.

getTemperatureHumidity();
getDS18B20();

Read all of the analog pins

  int Apin0 = analogRead(A0); // A0 is the photo resistor (light sensor)
  int Apin1 = analogRead(A1);
  int Apin2 = analogRead(A2);
  int Apin3 = analogRead(A3);
  int Apin4 = analogRead(A4);
  int Apin5 = analogRead(A5);
  int Apin6 = analogRead(A6);
  int Apin7 = analogRead(A7);
  thisTime = millis();

Create the payload as an array of integers, each item is two bytes long. Consisting of a Low byte and a High byte. The final integer result may be obtained from:

Low byte + 256* High byte
int send_payload[] = {Sensor, startTime, thisTime, Apin0, Apin1, Apin2, Apin3, Apin4, Apin5, Apin6, Apin7, temperatureC, humidity, DS18B20C};

The payload size for this generic data sensor payload is 28 bytes

int payload_size = sizeof send_payload;
radio.stopListening();

then send it

radio.write( send_payload , payload_size );

Download the sketch file and copy it to the Arduino working directory. Load the sketch into the Arduino IDE and then flash it to the Arduino sensor.

Step 10: Collect the Data and Feed the Form

Download the Python script RPi_logger5.py from Step 6. Rename the file to something more readable.

For example rpiforms.py.

Then from the Raspberry Pi navigate to the Python script downloaded.

Open a terminal then:

sudo python rpiforms.py

Start some sensors and watch the Raspberry Pi pull the data from the collected feeds and enter them into their respective google form fields. We may now view the data as it is collected from our Google Docs or Google Drive.

Here are the shared Links to the Form and the data sheet:

The Data Collection Form [Link].
The Data Viewing sheet [Link]

Step 11: Observations and Conclusions

Data visualization

We are collecting the data on one pipe and use Google sheets to split out each sensor from the Parent Form data and view it graphically. This allows quite rapid inference. For example the data collected here illustrates that the time intervals are perhaps a little too rapid. On the charts the data appears to have been collected at the same time interval, but has generated different temperature and humidity readings. In reality the readings are seconds apart,the chart scale covering the whole time frame of the data is giving this impression. However Google sheets allows manipulation of the Chart scales dependent on the display desired.

This data may be viewed and interrogated from almost anywhere. All that is required is access to the internet and Google Sheets.

Goals Met

The data is readily available from Google sheets

  1. Charts may be readily created
  2. Trends and patterns are easily observed
  3. Data may be readily shared
  4. The data posting method is relatively easy and does not require AUTH0 manipulation

Applications

  1. Laboratory Temperature, Humidity, Light, Bath Monitoring
  2. Home Automation, room monitoring
  3. Weather Station

Improvements

  1. Improve and clean the Python script
  2. Expand the nCurses Python script to scan more pipes
  3. Expand the nCurses Python script to create a dashboard menu allowing control of sensor or groups of sensors
  4. Introduce robust error handling
  5. Allow dynamic data handling

Improve and clean the Arduino sensor sketch

  1. Fix a standard sensor set and introduce more robust sensor routines to cover them
  2. Remove and clean up the Serial.print statements, a lot of them are there for debug
  3. The power saving routines are not implemented well, improve it, this will involve rethinking some of the sensors and hardware.
  4. The sensor identification is assigned at compile time, this is not the best way to perform this function, keep the sensor sketch generic, but allow the sensors to be assigned to/from eeprom.
  5. Overall polish for general deployment

Overall I have been able to monitor several sensors in real time using a very simple spread sheet. This has served my purpose extremely well.