Tag Archives: projects

Ambient Radio Weather Network

Nearly 7 years ago I started down a project to use existing oregon scientific weather sensors to collect temperature data throughout our house. The basic idea is that Oregon Scientific weather sensors all communicate unencrypted over 433Mhz wireless. With a receiver you can capture that data yourself and put it into other systems.

4th Generation of the Project

Over the last 7 years, and many iterations, lots has changed

  • Switched from heyu to directly talking to the rfxcom port in python
  • Moved from running on primary server to running on raspberry pi
  • Moved from storing data in mysql to publishing on MQTT bus
  • Abandoned the display layer entirely in favor of Home Assistant integration
  • Published the project on github and pypi

All of these have made the whole project a much more reasonable scope, and easier to understand what it is doing. So lets dive into some of the reasons for the more major ones.

Giving up on the Display Layer

When this project started the UI for it was a read only set of rrdtool generated graphs. One of the things you realize after a while is that while graphs are nice to understand trends, it's not enough. Min, Max, Current temperature is important, especially if you are using this information to understand tuning your HVAC system. How much differential is there between 2 points in the house right now. I started to imagine what the interface would have to look like to get all the data I wanted, and that became a giant visualization code base I never could get around to writing. But then along came Home Assistant.

Home Assistant is an open source home automation hub written in python. It already has a UI for displaying temperature sensors.

This also includes a detailed view:

While not perfect, that's a huge ton of work that I don't need to do any more. Yay! And better yet, by getting data into Home Assistant, these sensors can be used to trigger other parts of home automation. Even if that's just sending out an alert because a temperature went over a threshold.

MQTT

Ok, so now I am building a system with no UI, so the next question is how to get the data from this device into Home Assistant. For that I changed the architecture to being basically stateless and publishing data via MQTT.

MQTT is a lightweight message bus standard designed with IoT applications in mind. The protocol is pretty simple, there are multiple brokers that implement it, and there are client libraries for everything you can imagine (including arduino).

The ARWN project now is largely a relay that is blocked on the serial port reading data frames from the weather sensors and immediately publishing them out to MQTT

You can think of MQTT as a key / value bus. You publish on a topic, like 'arwn/temperature/Refrigerator' with an arbitrary payload. In my case I'm sending a json blob with all the relevant sensor data, as well as a timestamp. There is no timestamping inherent in MQTT, so if you care about when an event showed up, you have to insert the timestamp yourself in the payload.

I picked MQTT because Home Assistant already had very good support for it (while the primary message bus remains a python internal one, MQTT is strongly integrated in the project). Other open source projects like OpenHAB use MQTT as their primary bus. Bus architectures aren't a new thing, but the growth of the IoT space is making them even more relevant today.

There is code in Home Assistant now that will discover that that you've got ARWN running, and dynamically represent all the sensors it finds.

Even Better Dashboards

Home Assistant is limited by what it can store and what it can display. Fortunately it also can pump the data on it's bus into other systems as well. This include graphite and grafana.

Those are SVG by default and fully interactive, so you can mouse over points and get dropdowns about all the data at those points. It can be exported to PNG files as well.

Going forward

Since I started this project, there has been a whole revolution in software defined radio. For this project to be useful to people other than myself, the next step is to be able to pull the 433Mhz off an SDR  (which runs $20) instead of the rfxcom (which is about $120).

There are definitely pieces of Home Assistant integration to make better. One of which is to expose the rain data in Home Assistant at all. The other is a UX element to build a better way to visualize current wind / temp / rain in Home Assistant. That would be a new polymer component, which I haven't yet had time to dive into.

It's also been hugely valuable in the recent insulation work we got done, as I've got some data showing how effectively it changed the dynamics of the upstairs.

If you are interested in learning more, leave a comment/question, or checkout the code on github.

New Cold Frame

In the better late than never camp, I just finished off the new cold frame (an unheated green house). This is a project that started this summer, and I needed to finish it before first snow so that we can get the cars back in the garage.

Made of 2" Ceder and Thermoclear sheets (which is corrugated Lexan with a UV coat), it should hold up for quite a while.

DSC00041.JPG

DSC00042.JPG

DSC00043.JPG

 

All the joints were done with a Kreg Jig, a completely invaluable tool that I got to know during the day of service at the Poughkeepsie Farm Project a few years ago.

 

 

Weather Station Progress

(Note, this old post is background on some of the things I was doing here before.)

A few weeks ago I lost my rfxcom receiver during a lightning storm. Which meant my old weather station reporting on http://home.dague.net went defunct. My plan had already been to move away from that code (some incredibly hacky and fragile ruby) and do this over again less hacky in python on a raspberry pi. I'd even ordered and received a new rfxtrx receiver for this purpose. However, nothing gets you quite so motivated to make progress as downed infrastructure....

All through the week I've been getting data collection working and finally getting it uploaded to Weather Underground, where I'm personal weather station KNYPOUGH7.

I've even started to get some local graphing working as well, like the following:

test(outdoor temp, outdoor temp in cold frame, bedroom temp (sensor moved in the middle of the day for the spike), refrigerator temp, freezer temp)

The Raspberry Pi has made things interesting.... First, it's mobile, as I have it connected via wifi to the home network. This has been incredibly useful, as it turns out there is a sweet spot for seeing all my sensors consistently, which is only possible to get to if I don't have to worry about wired ethernet. The old solution had this sensor plugged directly into my home server, which meant drop outs on some of the sensors.

My old approach to data collection was rrdtool. As soon as I got a sensor reading I dumped it into an rrdtool database, which also made graphs. This had the advantage of being easy, and the disadvantage that rrdtool is a one way black hole for data. Once in, you can't ever get it out sanely. It's really designed to make operations centers look impressive and give admins the ability to eye trends, but makes working with that data not very useful. It does have the advantage of having a known fixed size, so you won't overrun your local storage.

In the new system I'm using mysql, on the pi. It's slow. Really slow. Often surprisingly slow. That slowness force one change in my thinking, which is to not use d3 to graph things client side, but instead generate static graphs with matplotlib on a cronjob, so user interaction won't be slow. It means the data might be a couple minutes behind, but for weather, that's fine.

The code... is currently terrible. However I've gone with the point of view that brute forcing working code right now is what I need to do to understand the patterns I want to build in this code. I spent too much time paralyzed on this project thinking through different ways to do it. So this will require some major refactoring and despecializing, already starting to happen, before it's of any value to anyone else. Once I get there, I'll have everything up on github as open source.

Finished Refinishing

These cedar benches were built by my father a number of years ago, after we took down a bunch of trees on the property, including some 100 year old cedar trees. After 3 years of weathering their original finish had worn through. Based on the success I had with our the cherry bench in the spring, I went to town on these this week.

IMAG0014.jpgThe one on the right is the state of the benches before being touched. The one on the left was after an hour with a belt sander. The cedar red color really jumped back out in the process.

IMG_1290.JPGThis is a look at the benches after the final coat of poly was put on. They are still drying, so there is some artifacting that won't be there once it hardens. I'm very curious how long the deep red heart wood is going to stay that color, or if it will orange like cedar does when exposed to air.

Start somewhere

When faced with a problem you don’t understand, do any part of it you do understand, then look at it again.

-Robert A. Heinlein, The Moon is a Harsh Mistress

I see a lot of folks that have lofty ideas, and never get anywhere, because they don't want to start until they are sure they can do it right the first time. Here's the thing, you can't do anything right the first time. So just dive in somewhere. Consider the mistakes lessons learned, and the cost of building expertise. Be prepared to feel stupid a lot, because that's the price of learning something new.

You'll never finish a project if you don't start it. You'll never complete a project, if you don't keep working on it. These may seem obvious, but it's impressive how many people don't seem to realize this.

(Thanks to Ed for posting this first.)

Reinvention

Some of the conversation that happened last night at the Hudson Valley Programmers gathering made me think about reinvention. There was a fundamental search engine problem that a few people were working with, which seemed like a solved problem, with good open source tools around it. When I asked the question, there was a lot of dismissal on those existing solutions as too slow. That could be true. It could also be that the existing solutions solve a lot of problems that they don't realize they've got to solve yet. Honestly, I don't know.

It raise and interesting general question, when is it appropriate to reinvent a solution to a solved problem? The answer isn't never, that leads to staleness in progress, as the solutions of old sometimes aren't appropriate any more. But the answer isn't always either, because that's just navel gazing with no forward progress. My experiences with Drupal have made me lean a little more towards less reinvention, if only because my time is precious, and if I'm going to do anything interesting in my hobby time I've got to build on top of the shoulders of giants. After building a dozen password reset systems over the years, I don't really find that that interesting.

Scratching an itch

Apparently I'm now writing a drupal drush module for patch management, and slowly understanding what that entails.  I've looked around and surprisingly there really isn't a good solution for this yet.  There are a number of solutions for applying other people's patches that are posted somewhere, but what I really care about is being able to easily keep, and reapply the dozen or so patches I've made to drupal modules to make mhvlug.org work.  Some of these were due to bugs that don't seem to be getting fixed any time soon.  Some are due to lost of drupal modules not working with PHP 5.3.

Regardless of the reason, I've apparently found a new itch to scratch, which hopefully isn't going to take me too long, because I really need to get back to android hacking.