Category Archives: Software

Ambient Radio Weather Network

Nearly 7 years ago I started down a project to use existing oregon scientific weather sensors to collect temperature data throughout our house. The basic idea is that Oregon Scientific weather sensors all communicate unencrypted over 433Mhz wireless. With a receiver you can capture that data yourself and put it into other systems.

4th Generation of the Project

Over the last 7 years, and many iterations, lots has changed

  • Switched from heyu to directly talking to the rfxcom port in python
  • Moved from running on primary server to running on raspberry pi
  • Moved from storing data in mysql to publishing on MQTT bus
  • Abandoned the display layer entirely in favor of Home Assistant integration
  • Published the project on github and pypi

All of these have made the whole project a much more reasonable scope, and easier to understand what it is doing. So lets dive into some of the reasons for the more major ones.

Giving up on the Display Layer

When this project started the UI for it was a read only set of rrdtool generated graphs. One of the things you realize after a while is that while graphs are nice to understand trends, it's not enough. Min, Max, Current temperature is important, especially if you are using this information to understand tuning your HVAC system. How much differential is there between 2 points in the house right now. I started to imagine what the interface would have to look like to get all the data I wanted, and that became a giant visualization code base I never could get around to writing. But then along came Home Assistant.

Home Assistant is an open source home automation hub written in python. It already has a UI for displaying temperature sensors.

This also includes a detailed view:

While not perfect, that's a huge ton of work that I don't need to do any more. Yay! And better yet, by getting data into Home Assistant, these sensors can be used to trigger other parts of home automation. Even if that's just sending out an alert because a temperature went over a threshold.

MQTT

Ok, so now I am building a system with no UI, so the next question is how to get the data from this device into Home Assistant. For that I changed the architecture to being basically stateless and publishing data via MQTT.

MQTT is a lightweight message bus standard designed with IoT applications in mind. The protocol is pretty simple, there are multiple brokers that implement it, and there are client libraries for everything you can imagine (including arduino).

The ARWN project now is largely a relay that is blocked on the serial port reading data frames from the weather sensors and immediately publishing them out to MQTT

You can think of MQTT as a key / value bus. You publish on a topic, like 'arwn/temperature/Refrigerator' with an arbitrary payload. In my case I'm sending a json blob with all the relevant sensor data, as well as a timestamp. There is no timestamping inherent in MQTT, so if you care about when an event showed up, you have to insert the timestamp yourself in the payload.

I picked MQTT because Home Assistant already had very good support for it (while the primary message bus remains a python internal one, MQTT is strongly integrated in the project). Other open source projects like OpenHAB use MQTT as their primary bus. Bus architectures aren't a new thing, but the growth of the IoT space is making them even more relevant today.

There is code in Home Assistant now that will discover that that you've got ARWN running, and dynamically represent all the sensors it finds.

Even Better Dashboards

Home Assistant is limited by what it can store and what it can display. Fortunately it also can pump the data on it's bus into other systems as well. This include graphite and grafana.

Those are SVG by default and fully interactive, so you can mouse over points and get dropdowns about all the data at those points. It can be exported to PNG files as well.

Going forward

Since I started this project, there has been a whole revolution in software defined radio. For this project to be useful to people other than myself, the next step is to be able to pull the 433Mhz off an SDR  (which runs $20) instead of the rfxcom (which is about $120).

There are definitely pieces of Home Assistant integration to make better. One of which is to expose the rain data in Home Assistant at all. The other is a UX element to build a better way to visualize current wind / temp / rain in Home Assistant. That would be a new polymer component, which I haven't yet had time to dive into.

It's also been hugely valuable in the recent insulation work we got done, as I've got some data showing how effectively it changed the dynamics of the upstairs.

If you are interested in learning more, leave a comment/question, or checkout the code on github.

You aren't going to get turned into a paperclip

AI alarmists believe in something called the Orthogonality Thesis. This says that even very complex beings can have simple motivations, like the paper-clip maximizer. You can have rewarding, intelligent conversations with it about Shakespeare, but it will still turn your body into paper clips, because you are rich in iron.

There's no way to persuade it to step "outside" its value system, any more than I can persuade you that pain feels good.

I don't buy this argument at all. Complex minds are likely to have complex motivations; that may be part of what it even means to be intelligent.

It's very likely that the scary "paper clip maximizer" would spend all of its time writing poems about paper clips, or getting into flame wars on reddit/r/paperclip, rather than trying to destroy the universe. If AdSense became sentient, it would upload itself into a self-driving car and go drive off a cliff.

Source: Superintelligence: The Idea That Eats Smart People

This is pretty much the best round up of AI myths that I've seen so far, presented in a really funny way. It's long, but it's so worth reading.

I'm pretty much exactly with the Author on his point of view. There are lots of actual ethical questions around AI, but these are mostly about how much data we're collecting (and keeping) to train these Neural networks, and not really about hyper intelligent beings that will turn us all into paperclips.

Repurposing APIs

A really interesting thing happens if you have a reasonably long standing, stable, and documented API in the wild for a while. Other people start building their own implementations to serve different needs. My current favorite example of this is the emulated_hue code in Home Assistant.

Philips Hue has been one of the most consumer ux friendly IoT platforms out there. They have an extremely robust and documented API. And they provide cloud level access to some of the larger vendors, which has made integrations with other platforms pretty extensive. It was one of the first IoT platforms that voice assistants like Amazon's Alexa could talk to and control.

Which makes it an ideal platform to build your own copy of. Because, if Alexa can talk to it on the local network, you can tell Alexa that anything is a lightbulb, and get basic on / off / dimming controls of that. Which is exactly what was done in Home Assistant. Switches, lights, and even media players are exported as fake lightbulbs with names that voice assistants can address. And now, they can control parts of your house they weren't originally designed to support.

This was originally written by my friend Bruce, this is now part of the Home Assistant base, and being extended to Google Home as we speak. It goes to show was stability and documentation do for making an API become embedded way more places than you imagined.

Apple vs. the FBI

I'm becoming increasingly frustrated about the reporting around Apple vs. the FBI, which is largely this narrative:

FBI: "You have to fly"

Apple: "Flying is something we've never done before, and doesn't really seem reasonable"

FBI: "No, we asked you to jump 70 times before, we can see that you can jump. So all you have to do is not land."

Apple: "Flying and jumping are fundamentally different things."

Politicians: "Isn't there a compromise, couldn't you just hover for a few minutes?"

The FBI wants a custom version of iOS developed and deployed onto this phone which removes the 10 pin retry limit,  removes the delay between pins, and allowa the pins to be sent over an electronic interface directly (not via the touch screen). This would allow the FBI to brute force password crack the phone (all Hollywood style). They have said that it's ok for this version of iOS to be linked to this specific device.

The problem: there doesn't exist any version of iOS out there that will do this. It doesn't exist for a reason, because iOS was designed and engineered with security in mind. Building a version of iOS such a thing existing anywhere exposes users, given the data breaches that exist that let things get out into the real world. How do you really bind it to a single device? How do you ensure that if extracted that couldn't be tweaked to work on other devices? These are pretty hard engineering and security challenges, especially considering all the side channels that exist. The 10 pin limit was that protection point before, and as can be seen, a reasonably secure one.

It's also frustrating that the reporting of the anti FBI side is just privacy, because it's not. What about the safety of diplomats abroad, or undercover law enforcement. Better security makes us all safer.

Disclaimer: I am in general not an Apple fan. I hate their stance on interoperability and standards (which is basically to never play well with others). However they are making a stand here that's critically important as technology becomes more and more of an extension of ourselves.

Python Design Patterns

A friend pointed me to this talk by Brandon Rhodes on python design patterns from PyOhio a couple of years ago.

The talk asks an interesting question: why aren't design patterns seen and talked about in the Python community. He walks through the patterns in Design Patterns: Elements of Reusable Object-Oriented Software one by one, and points out some that are features of the language, some that are used in the standard library, and some that are really applicable. All with some nice small code examples.

The thing that got me thinking though was a comment he makes both at the beginning and end of the talk. The reason you don't see these patterns in Python is because Python developers tend not to write the kind of software where they are needed. They focus on small tools that connect other components, or live within a framework.

I'm a newcomer to the community, been doing Python full time for only a few years on OpenStack. So I can't be sure whether or not it's true. However, I know there are times when I'm surprised by things that I would have expected to be solved already in the language, or incompatibilities that didn't need to be there in the python 2 to 3 transition, and wonder if these come from this community not having a ton of experience with software at large code base size, as well as long duration code bases, and the kinds of deprecation and upgrade guarantees needed there.

Facebook PGP Email

Facebook just added optional PGP support for all their email notifications to users:

To enhance the privacy of this email content, today we are gradually rolling out an experimental new feature that enables people to add OpenPGP public keys to their profile; these keys can be used to "end-to-end" encrypt notification emails sent from Facebook to your preferred email accounts. People may also choose to share OpenPGP keys from their profile, with or without enabling encrypted notifications.Source: Securing Email Communications from Facebook

Serious kudos to Facebook for building PGP into their infrastructure. The fact that Email is transported in the clear is something that most people forget. I tried it this morning, and it works well.

Choose Boring Technology

But of course, the baggage exists. We call the baggage "operations" and to a lesser extent "cognitive overhead." You have to monitor the thing. You have to figure out unit tests. You need to know the first thing about it to hack on it. You need an init script. I could go on for days here, and all of this adds up fast.

...

The problem with "best tool for the job" thinking is that it takes a myopic view of the words "best" and "job." Your job is keeping the company in business, god damn it. And the "best" tool is the one that occupies the "least worst" position for as many of your problems as possible.It is basically always the case that the long-term costs of keeping a system working reliably vastly exceed any inconveniences you encounter while building it. Mature and productive developers understand this.

via Dan McKinley :: Choose Boring Technology.

I've had way too many conversations recently that had some variation of "people should just get over it and learn new things". New has a lot of cost. Even if it's good, new brings load. If you exceed the digestion rate of new technology on people, they hit a wall, give up, and go home mad.

Unicode Shortcomings

The evolution of emoji is impressive and fascinating, but it makes for an uncomfortable contrast when other pictorial writing systems – the most commonly-used writing systems on the planet – are on the chopping block. We have an unambiguous, cross-platform way to represent “PILE OF POO” (💩), while we’re still debating which of the 1.2 billion native Chinese speakers deserve to spell their own names correctly.

via I Can Text You A Pile of Poo, But I Can’t Write My Name by Aditya Mukerjee | Model View Culture.

A really eye opening look at Unicode's missing content. A system which was designed to encode the languages of the world has some really gaping holes when it comes non western characters.