Tag Archives: weather

Ambient Radio Weather Network

Nearly 7 years ago I started down a project to use existing oregon scientific weather sensors to collect temperature data throughout our house. The basic idea is that Oregon Scientific weather sensors all communicate unencrypted over 433Mhz wireless. With a receiver you can capture that data yourself and put it into other systems.

4th Generation of the Project

Over the last 7 years, and many iterations, lots has changed

  • Switched from heyu to directly talking to the rfxcom port in python
  • Moved from running on primary server to running on raspberry pi
  • Moved from storing data in mysql to publishing on MQTT bus
  • Abandoned the display layer entirely in favor of Home Assistant integration
  • Published the project on github and pypi

All of these have made the whole project a much more reasonable scope, and easier to understand what it is doing. So lets dive into some of the reasons for the more major ones.

Giving up on the Display Layer

When this project started the UI for it was a read only set of rrdtool generated graphs. One of the things you realize after a while is that while graphs are nice to understand trends, it's not enough. Min, Max, Current temperature is important, especially if you are using this information to understand tuning your HVAC system. How much differential is there between 2 points in the house right now. I started to imagine what the interface would have to look like to get all the data I wanted, and that became a giant visualization code base I never could get around to writing. But then along came Home Assistant.

Home Assistant is an open source home automation hub written in python. It already has a UI for displaying temperature sensors.

This also includes a detailed view:

While not perfect, that's a huge ton of work that I don't need to do any more. Yay! And better yet, by getting data into Home Assistant, these sensors can be used to trigger other parts of home automation. Even if that's just sending out an alert because a temperature went over a threshold.

MQTT

Ok, so now I am building a system with no UI, so the next question is how to get the data from this device into Home Assistant. For that I changed the architecture to being basically stateless and publishing data via MQTT.

MQTT is a lightweight message bus standard designed with IoT applications in mind. The protocol is pretty simple, there are multiple brokers that implement it, and there are client libraries for everything you can imagine (including arduino).

The ARWN project now is largely a relay that is blocked on the serial port reading data frames from the weather sensors and immediately publishing them out to MQTT

You can think of MQTT as a key / value bus. You publish on a topic, like 'arwn/temperature/Refrigerator' with an arbitrary payload. In my case I'm sending a json blob with all the relevant sensor data, as well as a timestamp. There is no timestamping inherent in MQTT, so if you care about when an event showed up, you have to insert the timestamp yourself in the payload.

I picked MQTT because Home Assistant already had very good support for it (while the primary message bus remains a python internal one, MQTT is strongly integrated in the project). Other open source projects like OpenHAB use MQTT as their primary bus. Bus architectures aren't a new thing, but the growth of the IoT space is making them even more relevant today.

There is code in Home Assistant now that will discover that that you've got ARWN running, and dynamically represent all the sensors it finds.

Even Better Dashboards

Home Assistant is limited by what it can store and what it can display. Fortunately it also can pump the data on it's bus into other systems as well. This include graphite and grafana.

Those are SVG by default and fully interactive, so you can mouse over points and get dropdowns about all the data at those points. It can be exported to PNG files as well.

Going forward

Since I started this project, there has been a whole revolution in software defined radio. For this project to be useful to people other than myself, the next step is to be able to pull the 433Mhz off an SDR  (which runs $20) instead of the rfxcom (which is about $120).

There are definitely pieces of Home Assistant integration to make better. One of which is to expose the rain data in Home Assistant at all. The other is a UX element to build a better way to visualize current wind / temp / rain in Home Assistant. That would be a new polymer component, which I haven't yet had time to dive into.

It's also been hugely valuable in the recent insulation work we got done, as I've got some data showing how effectively it changed the dynamics of the upstairs.

If you are interested in learning more, leave a comment/question, or checkout the code on github.

Weather Station Progress

(Note, this old post is background on some of the things I was doing here before.)

A few weeks ago I lost my rfxcom receiver during a lightning storm. Which meant my old weather station reporting on http://home.dague.net went defunct. My plan had already been to move away from that code (some incredibly hacky and fragile ruby) and do this over again less hacky in python on a raspberry pi. I'd even ordered and received a new rfxtrx receiver for this purpose. However, nothing gets you quite so motivated to make progress as downed infrastructure....

All through the week I've been getting data collection working and finally getting it uploaded to Weather Underground, where I'm personal weather station KNYPOUGH7.

I've even started to get some local graphing working as well, like the following:

test(outdoor temp, outdoor temp in cold frame, bedroom temp (sensor moved in the middle of the day for the spike), refrigerator temp, freezer temp)

The Raspberry Pi has made things interesting.... First, it's mobile, as I have it connected via wifi to the home network. This has been incredibly useful, as it turns out there is a sweet spot for seeing all my sensors consistently, which is only possible to get to if I don't have to worry about wired ethernet. The old solution had this sensor plugged directly into my home server, which meant drop outs on some of the sensors.

My old approach to data collection was rrdtool. As soon as I got a sensor reading I dumped it into an rrdtool database, which also made graphs. This had the advantage of being easy, and the disadvantage that rrdtool is a one way black hole for data. Once in, you can't ever get it out sanely. It's really designed to make operations centers look impressive and give admins the ability to eye trends, but makes working with that data not very useful. It does have the advantage of having a known fixed size, so you won't overrun your local storage.

In the new system I'm using mysql, on the pi. It's slow. Really slow. Often surprisingly slow. That slowness force one change in my thinking, which is to not use d3 to graph things client side, but instead generate static graphs with matplotlib on a cronjob, so user interaction won't be slow. It means the data might be a couple minutes behind, but for weather, that's fine.

The code... is currently terrible. However I've gone with the point of view that brute forcing working code right now is what I need to do to understand the patterns I want to build in this code. I spent too much time paralyzed on this project thinking through different ways to do it. So this will require some major refactoring and despecializing, already starting to happen, before it's of any value to anyone else. Once I get there, I'll have everything up on github as open source.

New Temperature Sensors

I got my self some new temperature sensors for my hacked together home thermal monitor. The software that runs this is still aweful, that's on my project list. But at least now I'm able to figure out thermal variance inside the house, which is broader than I'd have expected.

A couple other changes were made in loading in the new sensors as well.

The outdoor temperature sensor is now on the North side of the house. This should minimize the amount of the day sunlight can hit the thing. While Oregon Scientific believes it can live in direct sunlight, my experience over the last two weeks is that it definitely can not. You get a 4 - 6 degree spike under direct sunlight. I could almost use the spikes to generate a map of the trees on the south side of the house based on when their shadows hit the sensor. The short of it, my reporting to Wunderground should be more accurate now.

The cold frame sensor is still at the far edge of receptivity, especially with the amount of earth it needs to go through. I build a tin foil reflector which seems to be helping a little, but it still drops out from time to time, generating the square waves in the curve. Not much I can do about that.

If you want to know more about this project, this post is probably the best starting point.

Mostly Cloudy or Partly Sunny?

A good question came in from a friend today about what is the difference between Mostly Cloudy and Partly Sunny.  While I had a general sense that it had to do with cloud percentages, I didn't actually know the details.  10 minutes on google, and I got to the Weather Service Operational Manual (WSOM) Chapter C-11, Zone and Local Forecasts - section 8.6. In a more readable format:

Term Expected Predominant or Average Sky Condition
Cloudy 9/10 to 10/10 opaque clouds
Mostly Cloudy or Considerable Cloudiness 7/10 to 8/10 opaque clouds
Partly Cloudy or Partly Sunny 3/10 to 6/10 opaque clouds
Mostly Clear or Mostly Sunny A Few Clouds and 1/10 to 3/10 opaque clouds
Clear or Sunny 0/10 to 1/10 opaque clouds

Opaque means you can't see the sun, moon, or stars through them (so wispy clouds don't count). Terms like "Fair" are specifically deprecated (though defined), and there is even a list of words that "are viewed as hedge terms and should be used sparingly and with great discretion".

Weather Channel on Android

I have to give the weather channel folks a lot of credit for their android application.  One of the quite nifty features is an alert model built into it that will actually translate national weather service alerts into phone alerts.  Today I got 2 of those.  One for the Tornado warning in the county, and the other for a very specific thunder cell moving over my location.  This is the second time that I've gotten an alert on an extremely strong thunder cell, and both times it came with a roughly 20 minute warning window.

Having my phone alert me when doom is coming is a quite good thing.  It also makes me wonder if in the future we'll have some new cellular version of the Emergency Broadcast System.  Given how few people are typically watching broadcast TV or Radio, something like this model would be quite slick.

Sniffing Oregon Scientific Weather Sensor Data

A few years ago I bought an Oregon Scientific wireless weather station.  It's a nice way to keep an eye on what's going on outside.  The unit supports up to 3 of these remote sensors (shown here), and aggregates it at a base station.  It doesn't have a computer interface, so while it displays nicely in the living room, I can't get access to that data.

Once upon a time I bought some 1-wire thermo sensors that I was going to wire that into my computer for data collection.  After the house was hit by lightning, I got far more gun shy about running conductive cables from the outside to a computer.

It occurred to me recently that there was good odds that someone must have figured out to sniff that wireless communication.  These aren't complicated devices, and Oregon Scientific seems to be letting different generations of sensors work with different generations of head units, implying that they have some pseudo standard protocol.  It would also solve my spark gap problem, as I could gather this data without any wires connected to the sensors.

After some googling I discovered that yes, these devices are operating in the 433 Mhz band, and yes, thanks to the folks at rfxcom, there is a unit out there which will receive this and spit it out in a reasonable usb interface.

Under Linux, this information can be very easily decoded with the heyu program.  This is really more of a command line system for home automation, but it also includes the rfxcom protocol and the decoders for just about everything Oregon Scientific makes.

After plugging in the rfxcom device, it took me about all of 30 minutes to get heyu compiled and configured for my devices.  The first time you run the heyu monitor you'll actually get big hex strings which are the raw data.  Once you tell heyu what kinds of sensors they are, you'll get it decoded in much more friendly units.  Here are the relevant lines in the heyu config file:

After configured and running, I now have sensor data being collected continuously:

The next steps here are going to be gathering this into something that I can use for graphing.  I've now got all the sensors I was looking for to be able to build my better thermostat brain.  Next steps will be tying this all together and starting my graphing.