All posts by Sean Dague

Microsoft's Inclusive Design Manual

From Microsoft's Inclusive Design Manual.

Microsoft's Inclusive Design website is pretty amazing. There is an overview manual, as well as exercises to help train yourself in inclusive design situations. However, even just reading the short gave me a few aha moments. It's worth the 30 minutes to give it a read through.

Kudos to Microsoft for both doing this work, and making it publicly available.

Over communicating

I once had a college class where the instructor was in the process of writing a text book for the class. So we were being taught out of photocopies of the draft textbook. It wasn't a very good class.

It wasn't that he wasn't a good writer. He was. The previous semester I'd had a great class using one of his text books. But it was taught by a different professor. There were some places that the text made a lot of sense to me, and some places where the different approach of the non-author professor made far more sense. With two points of view it's about synthesizing an understanding. If something from the book didn't really stick, something from in class might. And each aha moment made everything you'd read or heard before make a bit more sense.

I was reminded of this this morning reading through some technical documentation that was light on content and heavy on references. It was written that way so as to not repeat itself (oh, this part is explained over her instead). And while that may make sense to the authors, it doesn't make for an easy on ramp for people trying to learn.

It's fine to over communicate. It's good to say the same thing a few different ways over the course of a document, and even be repetitive at times. Because human brains aren't disk drives, we don't fully load everything into working memory, and then we're there. We pick up small bits of understanding in every pass. We slowly build a mental approximation of what's there. And people have different experiences that resonate with them, and that make more sense to them.

There isn't one true way to explain topics. So, when in doubt, over communicate.

 

In praise of systemd

There was definitely a lot of hate around systemd in the Linux community over the last few years as all the Linux distros moved over to it. Things change all the time, and I personally never understood this one. It wasn't like the random collection of just barely working shell scripts that may or may not have implemented "status" was really a technical superior solution.

Recently I've been plumbing more native systemd support into DevStack (OpenStack's development environment setup tool). Personally systemd is just another way to start services, it does a few things better at the cost of having to learn a new interface. But some huge wins in a developer context come from the new logging system with it, journald.

Systemd has it's own internal journaling system, which can reflect out to syslog and friends. However, it's inherently much richer. Journal messages can include not only text, but arbitrary metadata. You can then query the journal with this arbitrary metadata.

A concrete example of this is the request-id in OpenStack. On any inbound REST API call a request-id is generated, then passed around between workers in that service. This lets you, theoretically, trace through multiple systems to see what is going on. In practice, you end up having to build some other system to collect and search these logs. But with some changes to OpenStack's logging subsystem to emit journal native messages, you can do the following:

journalctl REQUEST_ID=req-3e4fa169-80ec-4635-8a7e-b60c16eddbcb

And get the following:

That, is the flow of creating a server in all the Nova processes.

That's the kind of analysis that most people were added Elastic Search into their environments to get. And while this clearly doesn't actually replicate all the very cool things you can do with Elastic Search, it does really give you a bunch of new tools as a developer.

Note: the enablement for everything above is still very much work in progress. The DevStack patches have landed, but are off by default. The oslo.log patch is in the proof of concept phase for review.

This idea of programs being able to send not just log messages, but other metadata around them, is really powerful. It means, for instance, you can always send the code function / line number where a message was emitted. It's not clogging up the log message itself, but if you want it later you can go ask for it. Or even if it is in the log message, by extracting in advance what the critical metadata is, you can give your consumers structured fields instead of relying on things like grok to parse the logs. Which simplifies ingest by things like Elastic Search.

There is a lot to explore here to provide a better developer and operator experience, which is only possible because we have systemd. I for one look forward to exploring the space.

Universal Design Problems

Credit: Amy Nguyen

A great slide came across twitter the other day, which rang really true after having a heated conversation with someone at the OpenStack PTG. They were convinced certain API behavior would not be confusing because the users would have carefully read all the API documentation and understood a set of caveats buried in there. They were also astonished by the idea that people (including those in the room) write software against APIs by skimming, smashing bits into a thing, getting one successful response, and shipping it.

The theme of the slide is really Empathy. You have to have empathy for your users. They know much less about your software then you do. And they have a different lived experience so even the way they would approach whatever you put out there might be radically different from what you expected.

Why Do Americans Refrigerate Their Eggs?

Americans love refrigeration, and eggs are high on the list of items we rush to get into the refrigerator after a trip to grocery store. Meanwhile, our culinary compatriots in Europe, Asia and other parts of world happily leave beautiful bowls of eggs on their kitchen counters.

So what gives?

Mostly, it’s about washing. In the U.S., egg producers with 3,000 or more laying hens must wash their eggs. Methods include using soap, enzymes or chlorine.

...

But — and here is the big piece of the puzzle — washing the eggs also cleans off a thin, protective cuticle devised by nature to protect bacteria from getting inside the egg in the first place. (The cuticle also helps keep moisture in the egg.)

With the cuticle gone, it is essential — and, in the United States, the law — that eggs stay chilled from the moment they are washed until you are ready to cook them. Japan also standardized a system of egg washing and refrigeration after a serious salmonella outbreak in the 1990s.

In Europe and Britain, the opposite is true. European Union regulations prohibit the washing of eggs. The idea is that preserving the protective cuticle is more important than washing the gunk off.

Source: ‘Why Do Americans Refrigerate Their Eggs?’ - The New York Time

It’s about 50 degrees warmer than normal near the North Pole, yet again - The Washington Post

Extreme temperature spikes such as this one have occurred multiple times in the past two winters, whereas they only previously occurred once or twice per decade in historical records according to research published in the journal Nature.

As Mashable science writer Andrew Freedman put it: “Something is very, very wrong with the Arctic climate.”

Source: It’s about 50 degrees warmer than normal near the North Pole, yet again - The Washington Post

As someone that follows the science, I definitely understand the difference between weather and climate. However, it takes climate change to create aberrations this extreme, this often.

This is all very real. It is unfortunate that many of our elected representatives don't agree with the science.

One of the largest icebergs recorded

The crack in Larsen C now reaches over 100 miles in length, and some parts of it are as wide as two miles. The tip of the rift is currently only about 20 miles from reaching the other end of the ice shelf.

Once the crack reaches all the way across the ice shelf, the break will create one of the largest icebergs ever recorded, according to Project Midas, a research team that has been monitoring the rift since 2014. Because of the amount of stress the crack is placing on the remaining 20 miles of the shelf, the team expects the break soon.

Source: A Crack in an Antarctic Ice Shelf Grew 17 Miles in the Last Two Months - The New York Times

Climate Change is real, and keeps on chugging. The visuals in the NY Times article are quite impressive and give you a more visceral sense of what is going on.

Gameplan for the next 4 years

Thing is: There is no someone else. No one is coming to save us from Trump and his merry band of egregious nincompoops. If there is saving to be done, it comes from us, or not at all. Be the “someone else” you want to see in this world. Because otherwise you’re leaving it to the horde of racists and bigots following in Trump’s wake. And that’s not acceptable.

At the very least, if you can’t get out of oh shit oh shit oh shit mode, then make goddamn sure you’re not making things harder for the people who are stepping up. I think it’s time to realize that we’re in a “perfect is the enemy of good” situation.

Source: The Beginning of the Trump Years – Whatever

Of all the post mortem writing of the election, I think that the science fiction authors seem to have the most sane grounding. Scalzi, Brin, Stross have all been far more interesting and insightful reads over the last 2 months than any of the punditry of what happened and what's next. You could do worse than follow those folks.

Scalzi's post today boiled down to a couple of pieces of solid advice for the next 4 years.

Stand up for one thing you believe in, and push on that. But remember not to overextend yourself. This is a marathon. There is just as much danger in getting overwhelmed in all the crazy and throwing your hands up and giving up.

Support others that are doing the same. No friendly fire that their cause isn't as important as yours. We make this world a better place by working in parallel.

The perfect is the enemy of the good. The point is to make progress, which you do one wandering and labored step at time.

What really causes cyber outages?

WASHINGTON, DC—For years, the government and security experts have warned of the looming threat of "cyberwar" against critical infrastructure in the US and elsewhere. Predictions of cyber attacks wreaking havoc on power grids, financial systems, and other fundamental parts of nations' fabric have been foretold repeatedly over the past two decades, and each round has become more dire. The US Department of Energy declared in its Quadrennial Energy Review, just released this month, that the electrical grid in the US "faces imminent danger from a cyber attack."

So far, however, the damage done by cyber attacks, both real (Stuxnet's destruction of Iranian uranium enrichment centrifuges and a few brief power outages alleged to have been caused by Russian hackers using BlackEnergy malware) and imagined or exaggerated (the Iranian "attack" on a broken flood control dam in Rye, New York), cannot begin to measure up to an even more significant cyber-threat—squirrels.

Source: Who’s winning the cyber war? The squirrels, of course | Ars Technica

The ultimate fuzz testers.

Ambient Radio Weather Network

Nearly 7 years ago I started down a project to use existing oregon scientific weather sensors to collect temperature data throughout our house. The basic idea is that Oregon Scientific weather sensors all communicate unencrypted over 433Mhz wireless. With a receiver you can capture that data yourself and put it into other systems.

4th Generation of the Project

Over the last 7 years, and many iterations, lots has changed

  • Switched from heyu to directly talking to the rfxcom port in python
  • Moved from running on primary server to running on raspberry pi
  • Moved from storing data in mysql to publishing on MQTT bus
  • Abandoned the display layer entirely in favor of Home Assistant integration
  • Published the project on github and pypi

All of these have made the whole project a much more reasonable scope, and easier to understand what it is doing. So lets dive into some of the reasons for the more major ones.

Giving up on the Display Layer

When this project started the UI for it was a read only set of rrdtool generated graphs. One of the things you realize after a while is that while graphs are nice to understand trends, it's not enough. Min, Max, Current temperature is important, especially if you are using this information to understand tuning your HVAC system. How much differential is there between 2 points in the house right now. I started to imagine what the interface would have to look like to get all the data I wanted, and that became a giant visualization code base I never could get around to writing. But then along came Home Assistant.

Home Assistant is an open source home automation hub written in python. It already has a UI for displaying temperature sensors.

This also includes a detailed view:

While not perfect, that's a huge ton of work that I don't need to do any more. Yay! And better yet, by getting data into Home Assistant, these sensors can be used to trigger other parts of home automation. Even if that's just sending out an alert because a temperature went over a threshold.

MQTT

Ok, so now I am building a system with no UI, so the next question is how to get the data from this device into Home Assistant. For that I changed the architecture to being basically stateless and publishing data via MQTT.

MQTT is a lightweight message bus standard designed with IoT applications in mind. The protocol is pretty simple, there are multiple brokers that implement it, and there are client libraries for everything you can imagine (including arduino).

The ARWN project now is largely a relay that is blocked on the serial port reading data frames from the weather sensors and immediately publishing them out to MQTT

You can think of MQTT as a key / value bus. You publish on a topic, like 'arwn/temperature/Refrigerator' with an arbitrary payload. In my case I'm sending a json blob with all the relevant sensor data, as well as a timestamp. There is no timestamping inherent in MQTT, so if you care about when an event showed up, you have to insert the timestamp yourself in the payload.

I picked MQTT because Home Assistant already had very good support for it (while the primary message bus remains a python internal one, MQTT is strongly integrated in the project). Other open source projects like OpenHAB use MQTT as their primary bus. Bus architectures aren't a new thing, but the growth of the IoT space is making them even more relevant today.

There is code in Home Assistant now that will discover that that you've got ARWN running, and dynamically represent all the sensors it finds.

Even Better Dashboards

Home Assistant is limited by what it can store and what it can display. Fortunately it also can pump the data on it's bus into other systems as well. This include graphite and grafana.

Those are SVG by default and fully interactive, so you can mouse over points and get dropdowns about all the data at those points. It can be exported to PNG files as well.

Going forward

Since I started this project, there has been a whole revolution in software defined radio. For this project to be useful to people other than myself, the next step is to be able to pull the 433Mhz off an SDR  (which runs $20) instead of the rfxcom (which is about $120).

There are definitely pieces of Home Assistant integration to make better. One of which is to expose the rain data in Home Assistant at all. The other is a UX element to build a better way to visualize current wind / temp / rain in Home Assistant. That would be a new polymer component, which I haven't yet had time to dive into.

It's also been hugely valuable in the recent insulation work we got done, as I've got some data showing how effectively it changed the dynamics of the upstairs.

If you are interested in learning more, leave a comment/question, or checkout the code on github.