Introduction: My Backyard Weather - Beta 2

By More of my stuff...
About: I'm a geek! I love learning about and working with all kinds of technologies. I am also a family man. I love spending time with family and friends. There is nothing better than letting the kids run around whil…

While building a weather station has been a "back of the head" interest for me since I was a kid, it wasn't until a few years ago when I started thinking seriously about practical home automation that I revisited the idea. If you want to automate common tasks that are easily forgotten, tedious, or need to happen when no one is home, you need data. A weather station can give you loads of useful data about the outside environment that can drive decisions in your system like "should I water the lawn?" or "someone should shut the kitchen window, it just started raining."

This project is one in a series of Home Automation projects I have in the works and unlike many of the gazillion other weather stations you'll find online, this project is meant to be chatty. It talks to other systems via a local database and a MQTT broker. It also talks to you the user via a web interface.

As the title suggests, this project is in "beta" state. In other words, it isn't finished. It does work as is, though and you can build and use this station to help control things in your home, now.

Step 1: Overview

What data does it provide?

This station collects most of the basic data you would expect from a typical station:

  • Temperature - What is the current temperature?
  • Humidity - What is relative humidity?
  • Dew Point - How saturated is the air?
  • "Feels Like" - A combination of wind chill and heat index, what temperature does it feel like outside?
  • Barometric Pressure - What is the current pressure in kPa?
  • Wind Speed / Direction - How fast is the wind blowing in mph and from what direction(deg)?
  • Rainfall (inches per interval) - How much has it rained in the last two minutes?
  • Rainfall (Is it raining) - Is it raining or not right now?
  • Lightning (counts per interval) - Is there lightning in the area and how intense is it?
  • LUX - How bright is it outside?
  • UV - What is the current levels of Ultraviolet Radiation outside?
    • UVA - How much Tanning / Damaging energy is there?
    • UVB - How much burning energy is there?

    • UV Index - What is the overall UV Index now?

In addition to the weather data above, I am also collecting data about the station itself for diagnostic use if there is an issue and to help improve the design. This data is stored in a separate table in the database.

  • Internal Temperature - How hot is it getting in the station enclosure?
  • Internal Humidity - How much humidity is in the enclosure?
  • Internal Dew Point - Is moisture about to condense on the electronics in the enclosure?
  • Lightning Disturbers - How much noise is the lightning detector having to filter? Could these be actual strikes?
  • DB Retries - Has the database failed to connect on the most recent attempt before succeeding?
  • DB Errors - How many errors were there since the system as started?
  • MQTT Errors - How many times was there an error attempting to publish station data?
  • DHT Sensor - Did the sensor start up properly?
  • THP Sensor - Did the sensor start up properly?
  • LightningSensor - Did the sensor start up properly?
  • LUXSensor - Did the sensor start up properly?
  • UVSensor - Did the sensor start up properly?
  • RTC - Did the Real Time Clock start up properly?
  • MINRAM - How low did the available RAM dip to?
  • AVGRAM - What is the average RAM value over an interval?
  • User Count - How many times did someone look at the station webpage over an interval?
  • Startup Timestamp - How long has the station been running?

Why reinvent the wheel? What did I gain?

There are hundreds of home brewed weather stations outlined on the Internet and dozens of well-made commercial stations available so why build one? There are two specific reasons - 1) to learn and 2) to get the data I needed and in a format I could control.

I learn best by doing. Since I don't have a background in weather science, I really had no idea where to begin and frankly, that's the best place to start. It makes you pay attention. For example, before I started researching this project, I thought that I had a grasp of what the "weather" was. However, I realized this was not the case. My awareness changed - rather than just listening to the weather channel waiting for the punchline (will it rain this afternoon), I now have a better understanding when they talk about topics like high and low pressure areas and what Dew Point represents. Learning this way makes you more aware, making the world both smaller and larger at the same time.

Additionally, none of the existing stations gave me control of the data in a way that I could depend on. I don't want to have to hack the equipment (if I have to hack it, then I might as well learn by building) and I didn't want to pull data off the cloud. This station reliably provides a data-set on a useful interval that is stored in a local database. I was able to choose what I wanted to measure against and while this station wasn't cheap, this system does have more options than most commercial systems of the same cost. Since this station doesn't rely on batteries, and is wired directly to my home network, I have been able to operate this station for more than two years uninterrupted without dropped packets or missing data.

Since the data is stored in a database, any number of scripts and applications can poke and prod the information for everything from "it's dark outside" to "based on the current conditions, the crawl vents should be closed." This data can also be used to find trends in energy consumption, irrigation patterns, and behavior to make decisions that can lower costs or improve the overall automation experience.

Lastly, I control the code of the station which, allows me to adapt to changes in my home automation system and add new features as needed. For example, I recently settled on an open home automation platform called Home Assistant. This software is a topic all it's own but for brevity, HA allows a myriad of inputs to control thousands of devices. One supported communication type is MQTT which is a fast publisher/subscriber system for managing messages. I added MQTT publishing of my data to the posting code of the weather station in an afternoon and had all of my weather data available to HA in minutes.

Sounds great, what's the downside?

Going this way isn't cheap. If you decide to build this project as I've described, it will set you back around $200 - $250 depending on how good you are at finding deals.

You're in for some headaches. Unlike buying a really good commercial weather station of comparable price, you aren't getting a meticulously tested and fool-proof system. This thing will make you upset till you get it right. After that, it works pretty well and will give you more data than you know what to do with.

That brings me to a third downside, manipulating the data is all on you. For some this is not a downside. For me it is a mixed bag. I'd like some help from time to time but overall, what I've learned is priceless.

Lastly, you need to be patient with a home built system as your time frame for testing is a full year. What I mean is, say you decide to change out a sensor for another one that appears to be a good fit and cheaper but will it stand up through different seasons. Unless you're able to build a simulation chamber, you'll just have to wait.

Step 2: Parts List

  • 1 - Arduino Mega 2560 - The primary processor. Others could be substituted but I've only tested the Mega.
  • 1 - Arduino Ethernet Shield - For wired network access.
  • 1 - DHT22 - This is the temperature and Humidity Sensor I am using to determine the environment inside the weather station case.
  • 1 - BME280 - This is the temperature /humidity / pressure sensor used for weather measurement.
  • 1 - SEN-08942 - Rain Gauge / Anemometer / Wind Vane
  • 1 - RainSensors.com RG-11 - Optical Rain Sensor for determining if it is currently raining or not.
  • 1 - TSL2561 - Luminosity Sensor (LUX) Sensor
  • 1 - MOD-1016- Lightning Detector
  • 1 - VEML6075 - Dual spectrum UV sensor
  • 1 - DS3231 - I2C Real Time Clock
  • 1 - E989NNJL - Weatherproof PVC New Work/Old Work Junction Box
  • 1 - 14252 - 1-Gang Gray Metal Weatherproof Standard Electrical Box
  • 1 - 1/2 in Connector - For connecting 1/2in conduit to the main enclosure
  • 2 - 1/2 in Connector Lock Nut - For securing connectors to main enclosure
  • 1 - 1/2 in Threaded Fitting- for connecting metal electrical box to main enclosure
  • 2 - Cable Connectors - Used to allow entry of instrument cables into the main enclosure
  • 1 - US-SA-AJD-198018 - 3.2"x4.3"x1.8" ABS Universal Project Enclosure w PC Transparent Cover used to hold the optical sensors
  • 1 - 30 x 30 x 1mm Quartz fused glass Sheet- Used for covering hole over UV sensor (UV transparent material)
  • 1 - PG-7 Cable Gland - Used to allow cable entry into the optical enclosure
  • 1 - RHRS - Solar Radiation Shield
  • 1 - B00NRHNPUA - Power Over Ethernet injection kit
  • 1 - 12V 1A DC Power supply - Very important - Make sure your power supply can supply 12v under load. See note below.
  • 1 - B019Q3U72M - 12V DC power supply
  • 1 - LM7809CV - 12VDC to 9VDC regulator
  • 1 - 330Ω resistor - RGB LED current limiting resistor
  • 1 - RGB LED - To show start up status
  • 2 - 10K Resistors - Pull up resistors for the DHT22 and Wind Vane.
  • 3 - RJ-12 Jacks - To connect the SEN-08942 cables and BME280 - optional.
  • 1 - Mega Proto Shield - To connect the instrumentation to the Mega
  • 1 - 1/2 Size Protoboard - For connecting optical and lightning sensors
  • 3 - 8pt Terminal Block - Instrumentation connectors
  • 2 - 6pt Terminal Block - Instrumentation connectors
  • 1 - 4pt Terminal Block - for RGB LED connector
  • Ethernet Cable - See details in the build instructions but this is for data/power and for short runs for component connections
  • 22 gauge wire - For connecing parts on proto boards
  • PCB Headers - For easy connection of the RTC and any experimental I2C components plus you'll need a 4 pin header for the wand.
  • 1 ft square of window screen - for covering any vent holes and for covering the wand top.
  • 1 - 10ft 1/2" metal conduit - This will serve as a mast for the anemometer and wind vein
  • Post or column - Used to mount station, the mast, the solar shield, and the optical sensor box

Step 3: Design

I've been working on a station now for a little over four years. The first year, though was mostly spent learning about the weather and what people were doing with stations. I then built several test platforms that helped me understand various communications schemes, power types, database updating methods and so on. As it turns out, there are loads of ways to make a station and not all of them are equal. Along the way I learned that solar and directly powered devices are about as equally reliable but wireless networking was not at all as reliable as wired. Some of the devices I chose were cheap and reliable and some were not. Ultimately, the design I wound up with has been very reliable for over a year.

The platform is powered by an Arduino Mega. I chose the Arduino line for simplicity since I preferred the dedicated nature of a MCU over a single board computer like a Raspberry Pi. I didn't want to have to troubleshoot the operating system in case I had a problem. Also the Mega, (chosen before the ESP line of processors became a thing) had the necessary resource requirements without breaking the bank. While it may be possible to squeeze down the code to fit on a Uno, as it stands I also needed more interrupt pins than the Uno supplied.

As stated above, I found that wireless networking would regularly fail* or require a restart from time to time. Plus, I wound up burning up a couple of wireless network cards for reasons I still don't understand. So, I opted for a wired approach which wasn't my preferred option. Since I had to run a cable anyway (and trying to find the "upside" of committing to all of that sweat equity burying a cable) I also opted to use power over Ethernet injectors over the much more expensive solar power option.

*It is worth noting here that I was having difficulty with wireless shields for the Arduino in an outdoor, electrically noisy, environment. I haven't tried to substitute an ESP8266 which has proven to be very reliable in other projects. The ESP could be swapped out for the Mega in this station with some code changes.

Along the way, I have used different sensors and settled upon the ones in the list above due to cost, durability, availability and usability. I didn't always buy the least expensive devices but looked at reviews of devices by others who had been using them for a while. I also wanted to use devices that were well documented and had code examples available.

Lastly, and this is important, I never planned on creating a station that is "accurate." My goals were to build a station that is moderately accurate but more importantly, consistent in it's readings. My back yard is not a place that will result in accurate readings due to terrain, trees and proximity to structures. So, my compromise is making sure that my data is useful for me (highly localized, consistent for my location, and reliable). I experimented with sealing/not sealing the primary enclosure, selecting the secondary transparent enclosure, placing sensors appropriately, and routing cables in a way that reduced noise. I am not concerned with contributing to outside weather data aggregation like weather underground (API is shut down anyway) or the weather channel.

Step 4: Putting It All Together

This build is actually pretty simple and straightforward. I look at the relatively simple circuit (attached) and I'm actually a little surprised. It hasn't felt that simple. :)

The core of this is the Arduino Mega. As stated before it wasn't chosen for the amount of pins it has but rather the number of interrupts and memory available. I needed 3 interrupts for the anemometer, lightning detector and rain gauge. Additionally, although it isn't really necessary, I'm also using an interrupt pin for the optical rain gauge. With the Mega, I took advantage of the the additional interrupt pin so that I could get away from polling the signal pin continuously.

Since I am writing to a database directly, I also need more RAM and program space than I could comfortably get away with in the UNO. The excellent MySQL library by Dr. Charles Bell, contains everything you need to read and write data but all of that functionality does come at a high cost - namely program space and operational RAM. It can run on an Uno but with everything I wanted to do, it wasn't really practical.

In my design, the pinout is as follows:

  • Pin A0 - Wind Vein analog input
  • Pin D2 - Lighning Interrupt Pin
  • Pin D3 - Anemometer Signal Pin
  • Pin D7 - DHT22 Internal Sensor Signal Pin
  • Pin D18 - Optical Rain Detector Signal Pin
  • Pin D19 - Tipping Bucket Rain Gauge Interrupt Signal Pin
  • Pin D20 - I2C SDA Pin
  • Pin D21 - I2C SCL Pin
  • Pin D46 - RGB LED Red Pin
  • Pin D48 - RGB LED Green Pin
  • Pin D50 - RGB LED Blue Pin

Post / Instrument Mounting Surface

I used a 4x4 pressure treated post for my instrument stand but you can use whatever surface is convenient keeping in mind the notes in the next section about location of instruments to other obstructions. I dug a trench out to my stand and buried a 1/2 inch PVC conduit. Inside the conduit was a single run of Ethernet plenum cable (solid core - not patch cable) so that I could terminate the end with a RJ-45 keystone jack. This was just my preference. If you want to spend the extra and run stranded cable end to end and terminate with a plug, that would work too.

One additional note is that in the event you're stand gets a nice jolt from a nearby lightning strike (not a direct hit as all bets are off in that event) you could also opt for fully shielded cable. You will want to run a ground strap from the cable to a strong ground surface or run a ground rod at the point of entry to your home - not at the stand as the potential difference between the two ground points could cause issues with your networking equipment.

Enclosures & Instruments

There are four enclosures in this project:

  1. Main enclosure - This houses the Arduino stack and the network / power connections. In my setup I have a small metal junction box attached to the side that contains the 12v power regulator. Since this component produces a lot of heat, it is attached to the metal cover with thermal adhesive allowing the cover to be a large heat-sink.
  2. Optical Enclosure - This enclosure houses the two optical sensors and the Lightning sensor. It is a sealed enclosure that has been modified for UV light.
  3. Solar Radiation Shield and 3D Printed Sensor Wand - The solar shield houses the BME280 in a 3D printed wand enclosure. This is detailed below.
  4. Optical Rain Detector - The optical sensor comes with it's own enclosure and wire gland.

Main Enclosure

The main enclosure is connected to 1/2 PVC buried conduit. This part is optional as you can also 1) run the wire over and aerial run, or 2) use a wireless transmitter and solar panel/battery to power the device. The conduit is connected to the main enclosure with a standard 1/2 in conduit connector and secured with a lock nut. Likewise, the power box is connected to the main enclosure with the threaded fitting and lock nut. To seal up the conduit fittings, I used a silicone calk.

The instrument cables enter the main enclosure using compression fittings but long cable glands would also work as long as the threaded end was around 1/2in long. If there are air gaps, seal them with silicone to keep bugs and moisture out.

NOTE: If you find that, despite your best efforts, you continue to get moisture build up inside the unit AND/OR if the internal temperature of the enclosure reaches 160° F, you can go the other route and drill an opening in the bottom of the enclosure and cover with window screen. Just keep in mind that if you live in a very cold winter environment, that the unit may not function without covering the opening.

Optical Enclosure

I have the LUX, UV and Lightning sensors all in a separate enclosure with a clear cover. The UV sensor needs to not be covered by the plexiglass. So, you will want to cut out a 3/4 in square over the UV sensor and cover it with a 2 in square of quartz glass sheet. This will allow the UV light to pass through and reach the sensor underneath. This is critical as most the case cover blocks UV light from passing through but you also don't want the UV light entering the LUX sensor either.

One additional note about the LUX sensor is that if it becomes saturated with light (begins sending back readings of 0 or 65K), you can optionally get a 1 in square piece of very light tint to place over the sensor. This will reduce the accuracy of the lux sensor in low light but will keep it from sending back nonsense as well.

To get the sensors as near the top of the enclosure as possible, I 3D printed a small block to raise the PCB up. I secured it with double sided tape. I then drilled a small hole for a wood screw in the middle of the PCB away from any traces that might cause a short.

If you don't want to print a block, you can double up on headers to bring the device up.

Solar Radiation Shield and 3D Printed Sensor Wand

The temperature / humidity / pressure sensor is a very small all-in-one package sensor called the BME280. I needed to place the sensor in the "open air" to get good readings but this sensor isn't suited for the outdoors. So, I have it in a solar radiation shield to protect it from the effects of direct sunlight / rain and I made a wand style enclosure with a screen to protect it from bugs and to hold it further up in the shield.

The 3D file for the pieces is here. (TinkerCad drawing)

To construct the wand:

  1. Print the three pieces of the wand from design on my Tinkercad profile linked above and at the end of this instructable. I used .2 layer height and 30% infill.
  2. Cut a piece of female pcb pin header down to 4 pins and shave off any extra plastic on the cut side so that it is smooth.
    Note: It helps if you cut the end so that the bottom is just a little bit wider than the top.
  3. Cut a cord to the length you need. Mine is about 4ft long. A 4 conductor phone cable makes for a convenient free cord that is often lying around the house.
  4. Slide the long printed base over your phone cord BEFORE soldering your cable onto the ens of the header pin.
  5. Solder on your header pins to the cable taking note which color is in which position.
  6. Slide the header into the printed center disc making sure that the top of the pins are exposed to the side of the disc that is narrowest. (see pictures).
  7. Before securing items in place, test your sensor to make sure that it works.
  8. Hot glue the header in place and hot glue around the header wires to act as strain relief.
  9. Glue the center disk to the long lower wand tube.
  10. Cut a piece of window screen about 6 inches long and 1 inch tall.
  11. Wrap the screen around the top of the sensor vent cover and glue it in place. The screen should wrap around the top twice. Make sure to tighten up the screen - if there are gaps around the top and bottom of the screen, bugs could get in and obscure the sensor.
  12. The top cover should snap into place on the center disc but if it is loose you can apply a small dot of hot glue to keep it from coming off. You don't want to fully glue to cover on as it is meant to allow you to change out the sensor if it fails.

The wand should slide up into the shield easily but if you find that the screen makes the top to big to get into the housing, then pull the mounting plate off of the shield and mount it in from the top instead and then reassemble the shield.

When you mount your solar shield to your instrument stand, you will want to place it on the north side of your post so
that it is a little more shielded from direct sunlight.

Optical Rain Sensor
The rain sensor uses a 12V supply voltage with ground and when "tripped" will close a normally open relay contact. For this contact, you'll need two wires. Although you'll only need 4 wires for this device, I used a standard Ethernet cord for this device as it's diameter was best suited for the cable gland. Connect your +12V wire to the P1 terminal block position and your ground wire to P2. Connect one of your two signal lines to the NO(normally open) terminal block position and the other to the COM position.

I have the settings for my rain gauge as follows - tipping bucket function, "on first detected drop, with a 15 minute delay from the last detected drop to an non-triggered state on the relay contacts and does not detect dark. For these settings the switch block with have the following switch states (order switches 1-8, 0=Off): 00101000. You can change these switches to get different behavior out of the sensor depending on your needs. Check the links section for a link to the documentation for this sensor.

Anemometer and Wind Vane

The wind instruments come with a handy mast assembly that can fit on the end of a 1/2 in piece of metal conduit. I suggest mounting the mast so that the instruments are at least 10 feet off of the ground for the best results. Keep in mind though that the cable from the instrument is also 10 foot so, you will need to take this length into consideration when running the line to your main enclosure.

The mast also comes with a rain gauge arm. Since total rainfall is best measured near the ground, I opted to use this arm instead to hold the optical rain sensor which, I simply strapped in place with two zip ties. You can also drill some small holes to secure it in place with long screws if you like.

Connections

NOTE: I recommend that you test all of your equipment prior to installing on your station. I especially recommend getting your wind vane directions fine tuned and oriented in the code before placing your mast. Once there, it is difficult to manipulate.

Pull your wires up through the connectors in the main enclosure. If your wires leave any gaps, you may want to use silicone to fill the gaps to keep bugs and moisture out. I used terminal blocks for the I2C sensor cables but you can also terminate to RJ-12 keystone jacks inside the main enclosure if you prefer. The jacks are handy when you need to plug or unplug for testing but they can be a liability over the long run due to corrosion or oxidation of the contact surfaces.

If you are using terminal blocks to secure wiring, I recommend you use a sharpie to mark which terminals connect what. You might remember now but will you remember a week or a month from now? Even after you mark them, take a picture of the blocks in case the markings fade over time.

Power

The optical rain sensor needs a solid 12 volts to work properly. Period.
If you've worked with wall warts at all, you have probably noticed by now that they hardly ever supply the rated voltage under load. In my case, a 12 volt supply I was using was actually supplying 8.5 volts under load. The Arduino and all of the other parts were happy with this result but my rain detector was not. It would trip and stay tripped until reset. So, I found one in my junk box that supplied 12.2volts under load reliably and used that instead. Problem solved, or so I thought - the mega would lockup - a lot. In increasing the voltage I introduced a new problem - heat. The onboard voltage regulator on the mega was good for voltages ranging from 6-20v but at 12volts, the onboard regulator was burning off that excess energy in the form of heat - a lot of it. When combined with a sealed enclosure, the system would overheat and lockup. My solution was to get a 12v to 9v power regulator and secure it with heat transfer adhesive to the lid of an external metal enclosure as described above.

You'll need to take the power coming off of the injector and split it. 12v runs directly to the VCC of the optical sensor and the other line route through the voltage regulator to the power supply. Make sure to tie all of your ground lines together.

Step 5: Mounting, Modifications, and Considerations

Location, Location, Location

So you'll need a location to place your instruments and the main enclosure. I used a pressure treated post that I already had from another project but you can use other surfaces. The one big note here is that location is very important.

One might expect that commercial back yard weather stations would be reasonably accurate and designed to be so but I've learned that this isn't the case. For example, wind measurement needs to happen at least 10 feet above the ground (professional measurement happens at 33ft) while rain measurement needs to happen about 2 feet off the ground. The reason is accuracy. Measuring each at those respective heights has the highest likelihood of an average mean measurement that is true. It isn't just a little off too, measuring rain at 10 feet can reduce accuracy as much as 10% and measuring wind speed below 10' can reduce accuracy by far more. So, the next time you see one of those all in one weather stations just know that you likely won't get accurate rainfall if it is place by the directions.

Also, your station needs to be out in the open and far away from other structures and trees. My back yard is in town and we have a lot of tall trees so, I centered mine as much as possible in the most open part of the yard however, I know that these obstacles reduce the amount of wind and rain my station measures. These measurements are accurate and pertinent to my home - just maybe not my neighbor's.

Tipping Bucket Rain Gauge

Cheap tipping buckets are well, cheap. Don't expect that they were well designed or accurate. My bucket had a too shallow lip around the edge of the collection area. I was getting errors of 25% - 40% depending on how hard it rained or how hard the wind was blowing. In short, the rain was splashing out of the collection funnel and not getting counted. I printed a riser and it stabilized the error.

Don't trust the paperwork. When the specifications say that each tip of the bucket will be 0.011 inches of rain, the real world results will be something different. In my case, the more accurate reading is .0138. This may be due to internal resistance of the center bar as it turns over the plastic notches.

Bigger is better. When it stops raining there is always a certain amount of rainfall that isn't heavy enough to tip the bucket so that amount isn't counted and evaporates. Larger buckets have smaller amounts of rainfall per tip so are more accurate in very light rain and missed buckets at the end of a shower.

With changes 1 and 2 listed above, rainfall measurements through this gauge is around 90% what is in my manual gauge. I'll take that.

Lightning Detector

I've had a lot of trouble out of the lightning detector. First it wouldn't read at all and this was due to a known manufacturer defect. Embedded Adventures, the vendor for this particular breakout board replaced it and for a while it seemed to work using a code library for the AS3935 chip. I discovered that it can be very touchy and incredibly sensitive to noise. For a while I was getting a lot of false positives and after a bit of crude analysis, I determined that it was too close to the wiring of the anemometer and picked up the events it was sending. I then moved it out to a separate box with the optical sensors.

I've easily spent far more time debugging the lightning sensor than I have any other sensor including the tipping bucket. These are tricky devices and I recommend patience when working with them. I also recommend reading the chip manufacturer datasheet.

Additionally, it may be worth your time to use a different vendor. I've had good customer service for this device from Embedded Adventures but since I started using this device, more vendors have brought the chip to market. Namely, Spark Fun has added two versions to their catalog.

Step 6: The Code

IDE Prep

The code for the weather station is attached but before you compile it, you'll need to add the library components. In the Arduino IDE, go to library manager and search for the following items:

  • DHT sensor library by Adafruit
  • Adafruit TSL2591
  • Adafruit Unified Sensor
  • Adafruit VEML6075
  • BME280 by Tyler Glenn
  • MySQL Connector Arduino by Dr. Charles Bell
  • PubSubClient by Nick O'Leary
  • RTCLib by NeiroN

Additionally you'll need to download and install the Lightning Sensor library from github here:
https://github.com/embeddedadventures/MOD-1016

Assumptions

The code makes some assumptions about your network and may need to be altered if these assumptions are false.

  • Your network has a DNS domain name defined
  • You have a MQTT broker and MySQL database locally served
  • Your database contains two fully built tables for data
  • Your network devices get handed IP addresses via DHCP
  • Your broker and mysql servers don't require ssl cert authentication

Overview

I've broken up the code into a series of files in the project folder. The general user configuration settings for server names, usernames and passwords are located in the userconfi.h file.

As this project is a work in progress, you're seeing the code as a snapshot in time. I've made numerous changes over the last months, weeks, days, so, I'll if you find a bug, please let me know. Soon I will replace this file with a github open project with a lot more open ended use cases, unit conversions, and options so, if interested, please stay tuned.

Setting Up Your Server

If you don't yet have a server on your network, this is a good time to setup one. Doing so is beyond the scope of this Instructable but to set you off in the right direction, here are some tips.

A "server" is only a computer on your network that can server a specific function to other computers. Any PC can be a server so, if you have an older one not being used, I would recommend starting there. Additionally, while you can use a raspberry pi as a server as many do for a host of reasons, it is not recommended as a database server. The number of writes to your SD card can kill it off pretty quickly depending on how frequently you get data from the station. For our use, an old desktop with a conventional hard drive is recommended.

You can install Ubuntu (fairly user friendly as Linux goes) and with a few quick commands, get MySQL and Mosquito installed and running. Here are a couple of videos to get things rolling:

Setting Up Your Database

This code doesn't currently build your database if it doesn't exist or is missing table structure. So, you'll need to build it by executing the following queries in your blank database.

Create a user. Change this script to use a preferred user name and password. NOTE: The username is shown on both lines:

CREATE USER 'my-mysql-db-user-name'@'localhost' IDENTIFIED WITH mysql_native_password AS 'my-mysql-db-user-password';
GRANT ALL PRIVILEGES ON *.* TO 'my-mysql-db-user-name'@'localhost' WITH GRANT OPTION;

Obviously, this is terrible security but if you're just starting out with a blank database, this will work.

Next create your database tables. If you want to change the structure you can do it now but just understand how those changes will affect the changes you'll also need to make to the insert command in the Arduino Sketch.

CREATE DATABASE IF NOT EXISTS `weather` DEFAULT CHARACTER SET latin1 COLLATE utf8_general_ci;
USE `weather`;

CREATE TABLE `interval_readings` (
  `temperature` float NOT NULL,
  `humidity` float NOT NULL,
  `dewpoint` float NOT NULL,
  `FeelsLike` float NOT NULL,
  `pressure` float NOT NULL,
  `wind_speed` float NOT NULL,
  `wind_direction` float NOT NULL,
  `WindOrdinalDirection` text NOT NULL,
  `rain_status` tinyint(1) UNSIGNED NOT NULL,
  `rainfall` float NOT NULL,
  `lightning_count` int(11) NOT NULL,
  `gusts` float NOT NULL,
  `lux` float NOT NULL,
  `uva` float NOT NULL,
  `uvb` float NOT NULL,
  `uvindex` float NOT NULL,
  `reading_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

CREATE TABLE `system_readings` (
  `temperature` float NOT NULL,
  `humidity` float NOT NULL,
  `dewpoint` float NOT NULL,
  `LightningDisturbers` int(11) NOT NULL,
  `DBRetries` int(11) NOT NULL,
  `DBErrors` int(11) NOT NULL,
  `MQTTErrors` int(11) NOT NULL,
  `DHTSensor` tinyint(1) UNSIGNED NOT NULL,
  `THPSensor` tinyint(1) UNSIGNED NOT NULL,
  `LightningSensor` tinyint(1) UNSIGNED NOT NULL,
  `LUXSensor` tinyint(1) UNSIGNED NOT NULL,
  `UVSensor` tinyint(1) UNSIGNED NOT NULL,
  `RTC` tinyint(1) UNSIGNED NOT NULL,
  `MINRAM` int(11) NOT NULL,
  `AVGRAM` int(11) NOT NULL,
  `UserCount` int(11) NOT NULL,
  `startup_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  `reading_timestamp` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

ALTER TABLE `interval_readings`
  ADD PRIMARY KEY (`reading_timestamp`);

ALTER TABLE `system_readings`
  ADD PRIMARY KEY (`reading_timestamp`);

One Full Cycle

It may be handy to think of this code as what happens over the course of a full reporting cycle. As it stands my reporting cycle is 2 minutes. I use this interval as a balance between too much data (shorter interval) and too slow a response time (longer interval). Depending on your needs, you can change this in the userconfig.h if you prefer.

Most of the time the station is idle. As the CPU cycles, it is waiting for 4 events - an interrupt to trigger, a interval timer (there are two), a website visitor, or a MQTT subscribed message to be received.

Interrupts - There are 4 interrupt triggers - wind, tipping bucket, lightning and rain first touching the optical rain sensor. When any of these events are triggered, different things happen based on the event. The optical sensor just sets a true/false bit. The lightning and tipping bucket events simply add to a running count.

The wind event has to keep track of a running wind speed as calculated by events / time. The event itself just counts out how much time has elapsed since the last event. Later calculations in the loop look for gusts and calculates an average over the current interval.

As I mentioned above, there are two timers in the code, the longer dataRecordTimer that sets how long to send out the station data and a shorter pollTimer used to check the wind direction and determine how much ram the system is using and average that out. This poll timer will be used in the future for some other house keeping tasks. Think of it as a house keeping timer.

When the system boots, it subscribes to a date/time message over MQTT. It uses this as a way to update the system clock and keep it in sync with the other devices on the network. In my home the message is published by my home automation system. If you don't want to send this message out on a regular interval, that's fine. The nice thing about MQTT is that you can subscribe to a message that no one is publishing. You can opt to manually send out a message from time to time to update your clock if you like. The message should look like:

environment/edate/%d-%m-%Y
environment/mtime/%H:%M:%S

Lastly, the system waits for web requests. If you open the device on port 80 in your browser, you'll see two tabs. The first with system statistics and the second showing the last posted weather data. This page should look the same across all screen sizes and browsers.

Attachments

Step 7: Integrations, Additional Notes, Links and Files

System Integration

One of the primary goals for building this station was an effort to use the data in home automation. I already outlined some use cases but currently, I am using my weather station for the following:

  • I am using light level data to automatically turn on the den lights when it gets dark enough outside. Although the home automation system can currently turn on the lights X amount of time before dark, we often need to override this if it is cloudy or stormy outside. The LUX data is really useful for this use case.
  • Our dog is terrified of storms and the real-time rain and lighting data is really helpful if we need to bring him in as a storm is starting up.
  • Other devices in our home display the weather data that is collected by this system. One of them, a bedside control panel is also listed here on Instructables.
  • I'm working on some data profiles to help with a home thermostat and an irrigation system I'm designing.

In the pictures above, I'm showing my weather station data in my Home Assistant interface. If you are also using Home Assistant, here are the sensor definitions:

- platform: mqtt
  state_topic: "weather/temperature"
  name: "Temperature"
  unit_of_measurement: "°F"
- platform: mqtt
  state_topic: "weather/pressure"
  name: "Pressure"
  unit_of_measurement: "mB"
  icon: mdi:download<
- platform: mqtt
  state_topic: "weather/humidity"
  name: "Humidity"
  unit_of_measurement: '%'
  icon: mdi:water-percent
- platform: mqtt
  state_topic: "weather/dewpoint"
  name: "Dew Point"
  unit_of_measurement: "°F"
- platform: mqtt
  state_topic: "weather/feelslike"
  name: "Feels Like"
  unit_of_measurement: "°F"
- platform: mqtt
  state_topic: "weather/windSpeed"
  name: "Wind Speed"
  unit_of_measurement: "MPH"
  icon: mdi:weather-windy-variant
- platform: mqtt
  state_topic: "weather/gusts"
  name: "Wind Gusts"
  unit_of_measurement: "MPH"
  icon: mdi:weather-windy
- platform: mqtt
  state_topic: "weather/windDirection"
  name: "Wind Direction"
  unit_of_measurement: "°"
  icon: mdi:compass
- platform: mqtt
  state_topic: "weather/windOrdinalDirection"
  name: "Wind Ordinal Direction"
  icon: mdi:compass
- platform: mqtt
  state_topic: "weather/rainfall"
  name: "Rainfall"
  unit_of_measurement: "IN"
  icon: mdi:weather-rainy
- platform: mqtt
  state_topic: "weather/lightning"
  name: "Ligtning Count"
  icon: mdi:weather-lightning
- platform: mqtt
  state_topic: "weather/lux"
  name: "Light Level"
  unit_of_measurement: "LUX"
- platform: mqtt
  state_topic: "weather/uva"
  name: "UVA"
- platform: mqtt
  state_topic: "weather/uvb"
  name: "UVB"
- platform: mqtt
  state_topic: "weather/uvindex"
  name: "UV Index"

Data Visualization

One advantage of storing your data in a local database is that you can use loads of different tools to analyze and visualize your data. Currently, I am using an open source tool called Grafana to create visualizations and dashboards. I highly recommend it as it is very easy to get started and is also quite powerful.

The JSON export of one of my simple Grafan dashboards is linked below.

Supporting Links

General Weather Resources:

Sensor Data Sheets and Code Examples:

Active Projects

This is one of several active projects I'm working on. If you'd like to know more about them or follow along with this one, you can do so on my website: https://andrewhoover.info/

Attachments

IoT Challenge

Second Prize in the
IoT Challenge