The 10,000 Year Clock Under Construction

A clock designed to ring once a year, for the next 10,000 years has begun installation in the mountains of west Texas. This is a project of the Long Now Foundation, a group dedicated to promoting long term thinking. Human civilization is roughly 10,000 years old, so lets think about what the next 10,000 years might bring.

Clock of the Long Now - Installation Begins from The Long Now Foundation on Vimeo.

I really love this clock. I love the hope that it represents. We have a lot of challenges to solve to get there, but setting a milestone like this puts a stake in the ground that we're going to fight to ensure there is someone hear it in 10,000 years.

The Long Now also has an excellent podcast / lecture series which you should add to your rotation.


MQTT, Kubernetes, and CO2 in NY State

Back in November we decided to stop waiting for our Tesla Model 3 (ever changing estimates) and bought a Chevy Bolt EV (which we could do right off the lot). A week later we had a level 2 charger installed at home, and a work order in for a time of use meter. Central Hudson's current time of use peak times are just 2 - 7pm on weekdays, and everything else is considered off peak. That's very easy to not charge during, but is it actually the optimal time to charge? Especially if you are trying to limit your CO2 footprint on the electricity? How would we find out?

The NY Independent System Operator (ISO) generates between 75% and 85% of the electricity used in the state at any given time. For the electricity they generate, they provide some very detailed views about what is going on.

There is no public API for this data, but they do publish CSV files at 5 minute resolution on a public site that you can ingest. For current day they are updated every 5 to 20 minutes. So you can get a near real time view of the world. That shows a much more complicated mix of energy demand over the course of the day which isn't just about avoiding the 2 - 7pm window.

Building a public event stream

With my upcoming talk at IndexConf next week on MQTT, this actually jumped up as an interesting demonstration of that. Turn these public polling data sets into an MQTT live stream. And, add some data calculation on top to calculate what the estimated CO2 emitted per kWh is currently. The entire system is written as a set of micro services on IBM Cloud running in Kubernetes.

The services are as follows:

  • ny-power-pump - a polling system that is looking for new published content and publishing it to an MQTT bus
  • ny-power-mqtt - A mosquitto MQTT server (exposed at It can be anonymously read by anyone
  • ny-power-archive - An mqtt client that's watching the MQTT event stream and sending data to influx for time series calculations. It also exposes recent time series as additional MQTT messages.
  • ny-power-influx - influx time series database.
  • ny-power-api - serves up a sample webpage that runs an MQTT over websocket bit of javascript (available at


MQTT is a light weight message protocol using a publish / subscribe server. It's extremely popular in the Internet of Things space because of how simple the protocol is. That lets it be embedded in micro controllers like arduino.

MQTT has the advantage of being something you can just subscribe to, then take actions only when interesting information is provided. For a slow changing data stream like this, giving applications access to an open event stream means being able to start doing something more quickly. It also drastically reduces network traffic. Instead of constantly downloading and comparing CSV files, the application gets a few bytes when it's relevant.

The Demo App

That's the current instantaneous fuel mix, as well as the estimated CO2 per kWh being emitted. That's done through a set of simplifying assumptions by looking at 2016 historic data (explained here, any better assumptions would be welcomed).

The demo app also includes an MQTT console, where you can see the messages coming in that are feeding it as well.

The code for the python applications running in the services is open source here. The code for the deploying the microservices will be open sourced in the near future after some terrible hardcoding is removed (so others can more easily replicate it).

The Verdict

While NY State does have variability in fuel mix, especially depending on how the wind load happens. There is a pretty good fixed point which is "finish charging by 5am". That's when there is a ramp up in Natural Gas infrastructure to support people waking up in the morning. Completing charging before that means the grid is largely Nuclear, Hydro, and whatever Wind is available that day, with Natural Gas filling in some gaps.

Once I got that answer, I set my departure charging schedule in my Chevy Bolt. If the car had a more dynamic charge API, you could do better, and specify charging once it flat lined at 1am, or dropped below a certain threshold.

Learn more at IndexConf

On Feb 22nd I'll be diving into MQTT the protocol, and applications like this one at IndexConf in San Francisco. If you'd love to discuss more about turning public data sets into public event streams with the cloud, come check it out.

Power usage after going Geothermal and EV

In November 2017 we replaced our Fuel Oil Heating system with a Geothermal one from Dandelion and bought a Chevy Bolt EV, which we're using as the primary car in the house. That for us means about 1000 miles a month on it. Central Hudson never actually read our meter in January, so applied an estimated based on our old usage. We finally got a meter reading, so now have a 2 month power usage that I can compare to the last couple of years.

By the Numbers

4700 kWh.

That seems like a lot, but I do have counters on both the furnace and the EV, which were ~2200 kWh and ~800 kWh respectively during this time period. Which leaves us at 1700 kWh for the rest of our load. That's compares to 1600 kWh last year, and 1500 kWh the year before.

There is also new electric load in the hot water system, which seems to be running pretty efficiently getting dumped waste heat from the water furnace.

This includes the stretch of time where we had a 14 day cold snap with 20 degree below average temperatures (ending with a record low). So while it's hard to compare to last year directly, it's pretty favorable. I'm sure that were we on oil we'd have had at least one tank fill during that window if not two, the oil trucks have been running pretty constant in the neighborhood.


Opening the power bill had a momentary "oh wow". But then realizing we no longer have an oil bill, and we've only paid for 1 or 2 tanks of gas in the Subaru in this window puts the whole thing in perspective.

Getting to a Zero Carbon Grid

This talk by Jesse Jenkins at UPENN is one of the best looks at what doing deep decarbonization of the grid really looks like. Jenkins is a PhD candidate at MIT researching realistic paths to get our electricity sector down to zero carbon emissions.

Price vs. Value

He starts with the common and simple refrain we all have, which is that research investments in solar have driven down the cost below that of fossil fuels, that cross over point has happened, and renewables will just take off and take over.

But that's the wrong model. Because of the intermitency of Wind and Solar, after a certain saturation point the wholesale value of a new MWh of their energy keeps decreasing. This has already been seen in practice in energy markets with high penetration.

 Sources of Energy

The biggest challenge is not all sources of energy are the same.

Jenkins bundles these into 3 categories. Renewables are great at Fuel savings, providing us a way not to burn some fuel. We also need a certain amount of fast burst on the grid, today this is done with Natural Gas Peaker plants, but demand hydro and energy storage fit that bill as well. In both of these categories we are making good progress on new technologies.

However, in the Flexible base camp, we are not. Today that's being provided by Natural Gas and Coal plants, and some aging Nuclear that's struggling to compete with so much cheap Natural Gas on the market.

How the mix changes under different limits

He did a series of simulations about what a price optimal grid looks like under different emissions limits given current price curves.

Under a relatively high emissions threshold the most cost efficient approach is about 40% renewables on the grid, some place for storage. The rest of the power comes from natural gas. 16% of solar power ends up being curtailed during the course of the year, which means you had to overbuild solar capacity to get there.

Crank down the emissions limit and you get more solar / wind, but you get a lot of curtailment. This is a 70% renewable grid. It's also got a ton of over build to deal with the curtailment.

But if you want to take the CO2 down further, things get interesting. 

Because of the different between price and value, relatively high priced Nuclear makes a return (Nuclear is a stand in for any flexible base source, it's just the only one we current have in production that works in all 50 states). There still is a lot of overbuild on solar and wind, and huge amounts of curtailment. And if you go for basically zero carbon grid, you get something a little surprising.

Which is the share of renewables goes down. They are used more efficiently, there is less curtailment. These are "cost optimal" projections with emissions targets fixed. They represent the cheapest way to get to a goal.

The important take away is that we're in this very interesting point in our grid evolution where cheap Natural Gas is driving other zero carbon sources out of business because we aren't pricing Carbon (either through caps or direct fees). A 40 - 60% renewables grid can definitely emerge naturally in this market, but you are left with a lot of entrenched Natural Gas. Taking that last bit off the board with renewables is really expensive, which means taking that path is unlikely.

But 100% Renewables?

This is in contrast to the Mark Jacobson 100% renewables paper. Jenkins points out that there have really been two camps of study. One trying to demonstrate the technical ability to have 100% renewables, the other looking at realistic pathways to zero carbon grid. Proving that 100% renewables is technically possible is a good exercise, but it doesn't mean that it's feasible from a land management, transmission upgrade, and price of electricity option. However none of the studies looking at realistic paths landed on a 100% renewables option.

Jenkins did his simulation with the 100% renewables constraint, and this is what it looked like.

When you pull out the flexible base you end up with a requirement for a massive overbuild on solar to charge sources during the day. Much of the time you are dumping that energy because there is no place for it to go. You also require storage at a scale that we don't really know how to do.

Storage Reality Check

The Jacobson study (and others) make some assumptions about season storage of electricity of 12 - 14 weeks of storage. What does that look like? Pumped hydro is currently the largest capacity, and most efficient way to store energy. Basically you pump water behind a dam when you have extra / cheap energy, then you release it back through the hydro facility when you need it. It's really straight forward tech, and we have some on our grid already. But scale matters.

The top 10 pumped hydro facilities combined provide us 43 minutes of grid power.

One of the larger facilities is in Washington state it is a reservoir 27 miles long, you can see it from space. It provides 3 1/2 minutes grid average power demand.

Pumped hydro storage is great, where the geography supports it. But the number of those places is small, and it's hard to see their build out increasing dramatically over time.

Does it have to be Nuclear?

No. All through Jenkins presentation Nuclear was a stand in for any zero carbon flexible base power source. It's just the only one we have working at scale right now. There other other potential technologies including burning fossil fuels but with carbon capture and storage, as well as engineered geothermal.

Engineered Geothermal was something new to me. Geothermal electricity generation today is very geographically limited you need to find a place where you have a geologic hot spot, and an underground water reserve, that's turning that into steam you can run through generators. It's pretty rare in the US. Iceland gets about 25% of it's power this way, but it has pretty unique geology.

However, the fracking technology that created the natural gas boom openned a door here. You can pump water down 2 miles into the earth and artificially create conditions to produce steam and harvest it. It does come with the same increase in seismic activity that we've seen in fracking, but there are thoughts on mitigation.

It's all trade offs

I think the most important take away is there is no silver bullet in this path forward. Everything has downsides. The land use requirements for solar and wind are big. In Jenkins home state of Massachusetts in order to get to 100% renewables it would take 7% of the land area. That number seems small, until you try to find it. On the ground you can see lots of people opposing build outs in their area (I saw a Solar project for our school district get scuttled in this way).

In the North East we actually have a ton of existing zero carbon energy available in Hydro Quebec, that's trapped behind not having enough transmission capacity. Massachusetts just attempted to move forward with the Norther Pass Transmission project to replace shutting the Pilgrim Nuclear facility, but New Hampshire approval board unanimously voted against it.

Vermont's shutdown of their Yankee Nuclear plant in 2014 caused a 2.9% increase in CO2 in the New England ISO region, as the power was replaced by natural gas. That's the wrong direction for us to be headed.

The important thing about non perfect solutions is to keep as many options on the table, as long as you can. Future conditions might change in a way where some of these options become more appealing as we strive to get closer to a zero carbon grid. R&D is critical.

That makes the recent 2018 budget with increased investment credits for Carbon Capture and Storage and small scale Nuclear pretty exciting from a policy perspective. These are keeping some future doors open.

Final Thoughts


Jenkins presentation was really excellent, I really look forward to seeing more of his work in the future, and for a wider exposure on the fact that the path to a zero carbon grid is not a straight line. Techniques that get us to a 50% clean grid don't work to get us past 80%. Managing that complex transition is important, and keeping all the options on the table is critical to getting there.

Python functions on OpenWhisk

Part of the wonderful time I had at North Bay Python was also getting to represent IBM on stage for a few minutes as part of our sponsorship of the conference. The thing I showed during those few minutes was writing some Python functions running in OpenWhisk on IBM's Cloud Functions service.

A little bit about OpenWhisk

OpenWhisk is an Apache Foundation open source project to build a serverless / function as a service environment. It uses Docker containers as the foundation, spinning up either predefined or custom named containers, running to completion, then exiting. It was started before Kubernetes, so has it's own Docker orchestration built in.

In addition to just the run time, it also has pretty solid logging and interactive editing through the webui. This becomes critical when you do anything that's more than trivial with cloud functions, because the execution environment looks very different than just your laptop.

What are Cloud Functions good for?

Cloud Functions are really good when you have code that you want to run after some event has occurred, and you don't want to maintain a daemon sitting around polling or waiting for that event. A good concrete instance of this is Github Webhooks.

If you have a repository that you'd like to do some things automatically on a new issue or PR, doing with with Cloud Functions means you don't need to maintain a full system just to run a small bit of code on these events.

They can also be used kind of like a web cron, so that you don't need a full vm running if there is just something you want to fire off once a week to do 30 seconds of work.

Github Helpers

I wrote a few example uses of this for my open source work. Because my default mode for writing source code is open source, I have quite a few open source repositories on Github. They are all under very low levels of maintenance. That's a thing I know, but others don't. So instead of having PR requests just sit in the void for a month I thought it would be nice to auto respond to folks (especially new folks) the state of the world.

Pretty basic, it responses back within a second or two of folks posting to an issue telling them what's up. While you can do a light weight version of this with templates in github native, using a cloud functions platform lets you be more specific to individuals based on their previous contribution rates. You can also see how you might extend it to do different things based on the content of the PR itself.

Using a Custom Docker Image

IBM's Cloud Functions provides a set of docker images for different programming languages (Javascript, Java, Go, Python2, Python3). In my case I needed more content then was available in the Python3 base image.

The entire system runs on Docker images, so extending those is straight forward. Here is the Dockerfile I used to do that:

This builds with the base, and installs 2 additional python libraries: pygithub to make github api access (especially paging) easier, and a utility library I put up on github to keep from repeating code to interact with the openwhisk environment.

When you create your actions in Cloud Functions, you just have to specify the docker image instead of language environment.

Weekly Emails

My spare time open source work mostly ends up falling between the hours of 6 - 8am on Saturdays and Sundays, which I'm awake before the rest of the family. One of the biggest problems is figuring out what I should look at then, because if I spend and hour figuring that out, then there isn't much time to do much that requires code. So I set up 2 weekly emails to myself using Cloud Functions.

The first email looks at all the projects I own, and provides a list of all the open issues & PRs for them. These are issues coming in from other folks, that I should probably respond to, or make some progress on. Even just tackling one a week would get me to a zero issue space by the middle of spring. That's one of my 2018 goals.

The second does a keyword search on Home Assistant's issue tracker for components I wrote, or that I run in my house that I'm pretty familiar with. Those are issues that I can probably meaningfully contribute to. Home Assistant is a big enough project now, that as a part time contributor, finding a narrower slice is important to getting anything done.

Those show up at 5am in my Inbox on Saturday, so it will be the top of my email when I wake up, and a good reminder to have a look.

The Unknown Unknowns

This had been my first dive down the function as a service rabbit hole, and it was a very educational one. The biggest challenge I had was getting into a workflow of iterative development. The execution environment here is pretty specialized, including a bunch of environmental setup.

I did not realize how truly valuable a robust Web IDE and detailed log server is in these environments. Being someone that would typically just run a vm and put some code under cron, or run a daemon, you get to keep all your normal tools. But the trade off of getting rid of a server that you need to keep patched is worth it some times. I think that as we see a lot of new entrants into the function-as-a-service space, that is going to be what makes or breaks them: how good their tooling is for interactive debug and iterative development.

Replicate and Extend

I've got a pretty detailed write up in the README for how all this works, and how you would replicate this yourself. Pull requests are welcomed, and discussions of related things you might be doing are as well.

This is code that I'll continue to run to make my github experience better. The pricing on IBM's Cloud Functions means that this kind of basic usage works fine at the free tier.

Slow AI

Charlie Stross's keynote at the 34th Chaos Communications Congress Leipzig is entitled "Dude, you broke the Future!" and it's an excellent, Strossian look at the future we're barelling towards, best understood by a critical examination of the past we've just gone through.

Stross is very interested in what it means that today's tech billionaires are terrified of being slaughtered by psychotic runaway AIs. Like Ted Chiang and me, Stross thinks that corporations are "slow AIs" that show what happens when we build "machines" designed to optimize for one kind of growth above all moral or ethical considerations, and that these captains of industry are projecting their fears of the businesses they nominally command onto the computers around them.

- Charlie Stross's CCC talk: the future of psychotic AIs can be read in today's sociopathic corporations

The talk is an hour long, and really worth watching the whole thing. I especially loved the setup explaining the process of writing believable near term science fiction. Until recently, 90% of everything that would exist in 10 years already did exist, the next 9% you could extrapolate from physical laws, and only really 1% was stuff you couldn't image. (Stross makes the point that the current ratios are more like 80 / 15 / 5, as evidenced by brexit and related upheavals, which makes his work harder).

It matches well with Clay Shirky's premise in Here Comes Everyone, that first goal of a formal organization is future existence, even if it's stated first goal is something else.

Do Demographics impact Leadership?

This morning on NPR there was a piece with Howard Dean about how older leaders should step aside and make way for a new generation. This has popped up in politics a bunch of times over the past couple of years, as the elected officials seem quite old. Our last presidential election gave options of a 69 and a 70 year old. With an ex president leaving office at the age of 55. The thing that most frustrates me about this reporting is it never talks about the root cause, which is demographics.

Birth Rates in the US 1943 - 2004: By Before My Ken - Wikipedia EN, Public Domain

After the baby boom, there was quite a baby bust. Births don't tell the whole story, as immigration / emigration still move population around, people die, etc. But here is a 2015 snapshot of population by age.

Population by age in the US (2015 snapshot)

You can see that birth shape still in overall demographics, including the 1970-71 mini spike. It's been filled in a bit by immigration, and the baby boom is less evident now that boomers are in their 60s and up and dying off. But it's still there. And it's important.

The Leadership Hole

About a decade ago I had the incredible opportunity to participate in a year long leadership excellence program at IBM. One of the classes was generational leadership, at a time when millennials were just about to enter the workforce. The teach talked about this demographic hole (all in generalizations).

The Baby Boomers were one of the largest generations, Gen X one of the smallest, the Millenials are nearly as big a generation as the Baby Boom. Gen X saw their parents laid off by large institutions, and the reaction is a huge upward trend in starting their own businesses. There are both less Gen Xers, and far less of them participating in large institutions still.

This means that as the Baby Boomers age out, who steps up? There aren't actually a ton of Gen Xers there in the wings. And the Millennials are all very young and haven't had the depth of experience yet. Do you try to fast promote them and hope they pick up enough along the way?  Many will, lots of "tribal knowledge" is going to be lost along the way. That might be good or bad depending on what habits were carried forward, but it's unlikely to be a smooth transition regardless.

What Generational Discontinuity Looks Like

Pew Research Center


Pew Research Center

This is what generational discontinuity looks like. A flip of 60 / 40 opinion of something being bad vs. good in a 10 year period. Conventional wisdom, accepted norms, flipping really quite quickly after decades of the old attitude being around without any real changes.

Through this lens, the "why now?" of the Silence Breakers are another one of these 60 / 40 flips that we are right in the middle of the crossover. All of this was "accepted" until it wasn't, and it will take a decade to fully entrench the new norm. Lots of people knew all this wasn't ok, but it took Millennials standing up, and Baby Boomers dying out, to flip it.

There are Costs

It's easy to see social progress and assume this is mostly upside. But there are costs of not carrying down experience and understanding. This analysis of the USS McCain collision showed how this leadership hole is hitting the Navy:

But the US Navy has its own particular history of creating fatigue through stress. The Navy's Surface Warfare community has a reputation for "eating its young." A "Zero-Defect" management mentality toward leadership has, despite entreaties from higher in the ranks, been ingrained in the surface Navy's command structure. When something happens, junior officers get burned. There is no learning from mistakes—only punishment for making them.

Couple that with tougher tasks than ever. Over the past decade, as a Government Accountability Office study in 2015 noted, "to meet the increasing demands of combatant commanders for forward presence in recent years, the Navy has extended deployments; increased operational tempos; and shortened, eliminated, or deferred training and maintenance." And to increase the availability of ships for ongoing missions in the Mediterranean and the Western Pacific, the Navy has "home-ported" ships overseas.

But the increased use of these forward deployed ships—which spent twice as much time at sea as similar ships based in the US—has had an impact on the training and maintenance of those ships in particular. In the Seventh Fleet, ships no longer even have a designated period for training. These days some Navy observers suggest the US may have the best equipped fleet in the world, but it no longer has the most skilled sailors driving them.

It's the same pattern. Extending the terms of older generation, younger generation not getting enough mentorship, and critical bits breaking down when younger generation are put on the spot without enough preparation.

Patterns in our Demographics

The predictive power of this demographic hole is important. Like a warming climate it's not going to tell you exactly which new super storm is going to hit which state. But it does tell us to expect an increase in:

  • Breakdowns in large systems / companies / organizations as the leadership hole prevented the pass down of unwritten rules and important tribal knowledge that kept the system working
  • Quick 60 / 40 opinion flips as the lack of gradual pass down of culture caused the younger generation to reevaluate their definition of normal

I just wish that more folks reporting on the events of the day pondered some of these larger factors and forcing functions of history like population demographics.

P.S. That's just from the disruption of flow of information through population bottle necks. If you start thinking about how opportunity exists when you hit the job market if you are in a dip or peak of the population, that gets fascinating as well. Malcolm Gladwell proposed that as part of the rise of the tech giants in Outliers. While he is always a bit light on research, it's interesting to ponder.


No Coal this Christmas Season - Personal Climate Action you can take now

"The best time to plant a tree was 20 years ago. The second best time is now."

Over the last year we've done a lot to our house to make it much less energy intensive, bought an electric car, and got  involved in Citizens' Climate Lobby. The research for all of that created a big pile of links for me, which I've tried to summarize here, to really show how many different ways you can make an impact.

This list is customized for New York, because that's where I live, and where I've done all my research. It would be great to see other folks build local guides for their areas as well, and I'd love to link to them.

Where Energy is Used

How do we use energy in the US? Because we measure electricity in kWh, gasoline / fuel oil in gallons, natural gas in cubic feet, sizing them all up and comparing them is hard. And we don't think about them as a single energy system. At a national level our energy is used by [1]:

  • Buildings - 40%
    • Residential - 20%
    • Commercial - 20%
  • Transportation - 28%
    • Cars, Light Trucks, Motorcycles - 16.2%
    • Other Trucks - 6.4%
    • Planes - 2.2%
  • Industry - 32%
    • Petroleum Refining - 10%
    • Chemical Production - 8.6%
    • Paper Production - 3.5%
    • Metals Production - 3%

The bits of this that always surprise me is that buildings are our key use of energy. Buildings are long term infrastructure. Our house was built in 1960, there are plenty of houses in our area build in 1900. Improving existing buildings is critical to making our infrastructure more efficient. Every improvement we've made over the last couple of years will live on beyond us in this home.

The other thing that sticks out is that we use 10% of our energy budget in the US refining petroleum. Much of that to be burned in other parts of the system. Every time we prevent a gallon of gas from burning, we don't only save it's emissions, but the emissions that happened when it was refined.



Average Home Energy use in NY State

Get a home energy inspection

In NY, the NY State Energy Research and Development Agency has many programs to increase energy efficiency. One of the programs is subsidized home energy audits to give you a targeted plan about what the biggest impacts for saving energy in your home will be.

Air sealing and Attic Insulation

Our home was built in 1960, and insulated to the standard at the time (which was not much). A year ago we went forward on our energy audit recommendations and got our attic air sealed, and 8" of cellulose insulation put on top. The results were dramatic. Heating dropped about 15%, my home office (which is the far end of the HVAC), no longer needed a space heater, and summer cooling was also dramatically reduced.

Get your energy inspection first, but realize that proper insulation in your home will dramatically, and immediately change the comfort level, and your energy use.

Replace your Oil Furnace with Geothermal

About 50% of homes in NY State heat with Fuel Oil. It is one of the dirtiest way to heat your home.

If you live in the Hudson Valley or Albany regions, Dandelion is a new geothermal company offering package deals to replace your existing oil system with a ground source heat pump. They put a well or two in your front yard, put a sealed tube down it, then use the 50 degree earth and a compressor to heat your home. Heat pumps get about 4 units of heat for every unit of electricity they consume. Ours has been in for about a month, and so far we're in love. So much quieter, no whiffs of oil smoke, and much more even distribution of heat in the house (it runs the fan slower and longer).

When I did the math, this was the single biggest climate impact we could make. This takes 700 gallons of fuel oil off the table. In comparison, we used about 500 gallons of gasoline in an average year between our cars.

Replace your Oil Furnace with... anything else

Seriously, Fuel Oil is terrible for the environment. While Natural Gas and Propane are still fossil fuels, they emit a lot less both in creating them, and when they burn. If you can't go the full hog to something like a heat pump, changing from Oil to NG or Propane will reduce your emissions on heating to about 1/2 of what they were before.


If you've not yet replaced all your lighting in your house with LEDs, do that now. They only use about 20% of the electricity of incandescent bulbs, are more efficient than even fluorescent, and last an incredibly long time (25 year lifespans are common).

If you are a Central Hudson customer you can get 60W replacement bulbs for $1 each. Just do it. While lighting use is overall a pretty low part of your energy budget, it is also very actionable if you haven't done the conversion to LEDs yet. And, LED lights fit in christmas stockings.


The path to decarbonizing the economy is to electrify everything, while simultaneously making the electric grid less carbon intensive.


NY State's energy production is relatively low carbon, but if we are going to fully decarbonize we do need to reduce natural gas consumed for electricity as much as possible.

Choose a Green ESCO

NY State allows you to choose your energy producer (energy services company, or ESCO). There are a number of companies that provide you with energy from wind farms that they are building regionally. This typically mean a small rise in your energy costs, but that comes with supporting the build out of new renewables.

Two good options in our area are:

Community Solar

NY State has new rules in place that allow for Community Solar in our area. These are small scale (2 Mega Watt or less) facilities that you can sign up and get your power from solar even if you can't put solar panels on your roof (you have bad site, or are a renter).

Solarize Hudson Valley has sign up information for folks in the area. If you are in the Central Hudson power generation region, Nexamp is building a facility in Wappingers Falls. We've signed up, and starting in May of 2018, will be getting our power from solar.

Carbon Offsets

If there is nothing on the list that works for you, but you still want to have an impact on reducing your carbon footprint, consider some kind of carbon offset. Carbon offset projects work to capture carbon, or reduce emissions from something like a landfill. We all share one atmosphere, so any way you reduce emissions helps.

The carbon offset market is a wildly confusing place as an individual. As a NY (or North East) resident, the Carbon Reduction Certificates from the Adirondack Counsil is great. Each certificate is used to buy 1 ton of CO2 off the Regional Greenhouse Gas Initiative annual carbon auction (a carbon trading system for power companies that 9 states have agreed to, and NJ and VA might be joining soon). The price for a ton of carbon on the RGGI is still pretty low, so left over proceeds go to their micro grants program which support local energy efficiency and emissions reduction.


Get Engaged

Right now, you need to do something extra, or out of the ordinary to have an impact on climate change. Citizens' Climate Lobby is a political action group looking to change that, by pricing carbon in the economy. A real price on carbon would make doing the efficient thing, also be doing the cheaper thing, which would make it the default choice in most situations.

We've got a local chapter that meets in Beacon, NY once a month, and so if you want to flex your political muscles, as well as your economic ones, sign up and join us.

It all maters

Every action you make matters. And the exciting fact is that there are so many things you can do now to have an impact (including many things not on this list). So take a minute this holiday season and think about how you can take a little bit of coal out of your own Christmas season.


Syncing Sieve Rules in Fastmail, the hard way

I've been hosting my email over at Fastmail for years, and for the most part the service is great. The company understands privacy, contributes back to open source, and is incredibly reliable. One of the main reasons I moved off of gmail was their mail filtering system was not fine grained enough to deal with my email stream (especially open source project emails). Fastmail supports sieve, which lets you write quite complex filtering rules. There was only one problem, syncing those rules.

My sieve rules are currently just north of 700 lines. Anything that complex is something that I like to manage in git, so that if I mess something up, it's easy to revert to known good state.

No API for Sieve

Fastmail does not support any kind of API for syncing Sieve rules. There is an official standard for this, called MANAGESIEVE, but the technology stack Fastmail uses doesn't support it. I've filed tickets over the years that mostly got filed away as future features.

When I first joined Fastmail, their website was entirely classic html forms. Being no slouch, I had a python mechanize script that would log in as me, then navigate to the upload form, and submit it. This worked well for years. I had a workflow where I'd make a sieve change, sync via script, see that it generated no errors, then commit. I have 77 commits to my sieve rules repository going back to 2013.

But, a couple of years ago the Fastmail team refreshed their user interface to a Javascript based UI (called Overture). It's a much nicer UI, but it means it only works with a javascript enabled browser. Getting to the form box where I can upload my sieve rules is about 6 clicks. I stopped really tweaking the rules regularly because of the friction of updating them through clear / copy / paste.

Using Selenium for unintended purposes

Selenium is pretty amazing web test tool. It gives you an API to drive a web browser remotely. With recent versions of Chrome, there is even a headless chrome driver, so you can do this without popping up a graphics window. You can drive this all from python (or your language of choice).

An off hand comment by Nibz about using Selenium for something no one intended got me thinking: could I manage to get this to do my synchronization?

Answer, yes. Also, this is one of the goofiest bits of code that I've ever written.

Basic Flow

I won't do a line by line explanation, but there are a few concepts that make the whole thing fall in line.

The first is the use of WebDriverWait. This is an OvertureJS application, which means that clicking parts of the screen trigger an ajax interaction, and it may be some time before the screen "repaints". This could be a new page, a change to the existing page, an element becoming visible. Find a thing, click a thing, wait for the next thing. There is a 5 click interaction before I get to the sieve edit form, then a save button click to finish it off.

Finding things is important, and sometimes hard. Being an OvertureJS application, div ids are pretty much useless. So I stared a lot in Chrome inspector at what looked like stable classes to find the right things to click on. All of those could change with new versions of the UI, so this is fragile at best. Some times you just have to count, like finding the last textarea on the Rules page. Some times you have to inspect elements, like looking through all the buttons on a page to find the one that says "Save".

Filling out forms is done with sendKeys, which approximates typing by sending 1 character every few milliseconds. If you run non headless it makes for amusing animation. My sieve file is close to 20,000 characters, so this takes more than a full minute to put that content in one character at a time. But at least it's a machine, so no typos.

The Good and the Bad

The good thing is this all seems to work, pretty reliably. I've been running it for the last week and all my changes are getting saved correctly.

The bad things are you can't have 2 factor enabled and use this, because unlike things like IMAP where you can provision an App password for Fastmail, this is really logging in and pretending to be you clicking through the website and typing. There are no limited users for that.

It's also slow. A full run takes

It's definitely fragile, I'm sure an update to their site is going to break it. And then I'll be in Chrome inspector again to figure out how to make this work.

But, on the upside, this let me learn a more general purpose set of tools for crawling and automating the modern web (which requires javascript). I've used this technique for a few sites now, and it's a good technique to add to your bag of tricks.

The Future

Right now this script is in the same repo as my rules. This also requires setting up the selenium environment and headless chrome, which I've not really documented. I will take some time to split this out on github so others could use it.

I would love it if Fastmail would support MANAGESIEVE, or have an HTTP API to fetch / store sieve rules. Anything where I could use a limited app user instead of my full user. I really want to delete this code and never speak of it again, but a couple of years and closed support tickets later, and this is the best I've got.

If you know someone in Fastmail engineering and can ask them about having a supported path to programatically update sieve rules, that would be wonderful. I know a number of software developers that have considered the switch to Fastmail, but stopped when the discovered that updating sieve can only be done in the webui.

Updated (12/15/2017): via Twitter the Fastmail team corrected me that it's not Angular, but their own JS toolkit called OvertureJS. The article has been corrected to reflect that.


REST API Microversions

This is the version of the talk I gave at API Strat back in November about OpenStack's API microversions implementation, mostly trying to frame the problem statement. The talk was only 20 minutes long, so you can only get so far into explaining the ramifications of the solution.


This talk dives into an interesting issue we ran into with OpenStack, an open source project that is designed to be deployed by public and private clouds, and exposes a REST API that users will consume directly. But in order to understand why we had to do something new, it's first important to understand basic assumptions on REST API versioning, and where those break down.


There are some generally agreed to rules with REST API Versioning. They are that additive operations, like adding a key, shouldn't break clients, because clients shouldn't care about extra data that they get back. If you add new code that adds a new attribute, like description, you can make these changes, roll them out to the user, they get an extra thing and life is good.

This mostly work as long as you have a single instance, so new attributes show up "all at once" for users, and that you don't rollback an API change after it's been in production for any real length of time.


This itself is additive, you can keep adding more attributes over time. Lots of public web services use this approach. And this mostly just works.


It's worth thinking for a moment about the workflow that supports this kind of approach. There is some master branch where features are being written and enabled, and at key points in time these features are pushed to production, where those features show up for users. Basic version control, nothing too exciting here.


But what does this look like in an Open Source project? In open source we've got a master upstream branch, and perhaps stable releases that happen from time to time. Here we'll get specific and show the OpenStack release structure. OpenStack creates a stable release every 6 months, and main development continues on in git master. These are named with letters of the alphabet, so Liberty, Mitaka, Newton, Ocata, Pike being the last 5 releases.


But releasing stable code doesn't get it into the hands of users. Clouds have to deploy that code. Some may do that right away, others may take a while. Exactly when a release gets into the users hands is an unknown. This gets even more tricky when you consider private clouds that are consuming something like a Linux Distro's version of OpenStack. There is a delay getting into the distro, then another delay about when they decide to upgrade.


End users have no visibility into the versions of OpenStack that operators have deployed, they only know about the API. So when they are viewing the world across clouds at a specific point in time (T1, T2, or T3) they will experience different versions of the API.


Let's take that T3 example. If a user starts by writing their software to Cloud A, the description field is there. They are going to assume that's just part of the base API. They then connect their software to Cloud B, and all is fine. But, when later in the week they point it at Cloud C, the API has magically "removed" attributes. Removing attributes in the server is never considered safe.

This might blow up immediately, or it might just carry a null value that fails very late in the processing of the data.


The lesson here is that the assumed good enough rules don't account for a different team developing software than deploying it. Sure, you say, that's not good Dev Ops, of course it's not supported. But be careful with what you are saying there, because we deploy software we don't develop all the time, 3rd party open source. And if you've come down firmly that open source is not part of Dev Ops, I think a lot of people would look at you a bit funny.


I'm responsible for two names that took off in OpenStack, one is Microversions. Like any name that catches on you regret it later because it misses the important subtlety of what is going on. Don't ask me to name things.

But besides the name, what are microversions?


Let's look at that example again, if we experience the world at time T3, with Clouds A, B, and C, the real issue is that hitting Cloud C appears to make time "go backwards". We've gone back in time and experienced the software at a much earlier version. How do we avoid going backwards?


We introduce a header for the "microversion" we want. If we don't pass one, we get the minimum version the server supports. Everything after that we have to opt into. If we ask for a microversion the server doesn't support we get a hard 400 fail on the request. This lets us fail early, which is more developer friendly than giving back unexpected data which might corrupt things much later in the system.


Roughly, microversions are inspired by HTTP content negotiation, where you can ask for different types of content from the same url, and the server will give you the best it can (you define "best" with a quality value). Because most developers implementing REST clients aren't deeply knowledgeable about HTTP low level details, we went for simplicity and did this with a dedicated header. For simplicity we also make this a globally incrementing value across all resources, instead of per resource. We wanted there to be no confusion what version 2.5 was.

The versions that a service supports are discoverable by hitting the root document of the API service. The other important note is that services are expected to continue to support old versions for a very long time. In Nova today we're up to about 2.53, and we still support everything back down to 2.0. That represents about 2.5 years of API changes.

There are a lot more details on the justification here for the approach, but not enough time today to go into them. If you want to learn more, I've got a blog writeup from when we first did that that dives in pretty deep, including showing the personas you'd expect to interact with this system.


Thus far this has worked pretty well. About 1/2 the base services in OpenStack have implemented a version of this. Most are pretty happy with the results. That version document I talked about can be seen here.

There are open questions for the future, mostly around raising minimum versions. No one has done it yet, though there are some thoughts about how you do that.


Since I'm here, and there are a lot of OpenAPI experts around, I wanted to talk just briefly about OpenStack and OpenAPI.


The OpenStack APIs date back to about 2010, when the state of the art for doing REST APIs was WADL. A now long dead proposed specification by Sun Microsystems. Lots of XML. But, that was the constraints at the time, which are different constraints than OpenAPI.

One of those issues is our actions API, where we use the same url with very different payloads to do non-RESTy function calls. Like reboot a server. The other is Microversions, which don't have any real way to make to OpenAPI without using vendor extensions, at which point you loose most of the interesting tooling in the ecosystem.

There is an open question in my mind about whether the microversion approach is interesting enough that it's something we could consider for OpenAPI. OpenStack could easily microversion itself out of the actions API to something more OpenAPI friendly, but without microversion support there isn't much point.


There was a talk yesterday about "Never have a breaking API change again", which followed about 80% of our story, but didn't need something like microversions because it was Azure, and they controlled when code got deployed to users.

There are very specific challenges for Open Source projects that expose a public web services API, and expect to be deployed directly by cloud providers. We're all used to open source behind the scenes, and plumbing to our services. But Open Source is growing into more areas. It is our services now. With things like OpenStack, Kubernetes, OpenWhisk... Open Source projects now are defining that user consumable API. If we don't come up with common patterns for how to handle it then we're putting Open Source at a disadvantage.

I've been involved in Open Source for close to two decades, and I strongly believe we should make Open Source able to play on the same playing field as proprietary services. The only way we can do this is think about if our tools and standards support Open Source all the way out to the edge.


Question 1: How long do you expect to have to keep the old code around, and how bad it is to manage that?

Answer: Definitely a long time, minimum a few years. The implementation of all of this is in python and we've got a bunch of pretty good decorators and documentation that makes it pretty easy to compartmentalize the code. No one has yet lifted a minimum version, as the amount of work to support the old code hasn't really been burdensome as of yet. We'll see how that changes in the future.

Question 2: Does GraphQL solve this problem in a way that you wouldn't need microversions?

Answer: That's a very interesting question. When we got started GraphQL was pretty nascent so not really on our radar. I've spent some time looking at GraphQL recently, and I think the answer is "yes, in theory, no in practice", and this is why.

Our experience with the OpenStack API over the last 7 years is no one consumes your API directly. They almost always use some 3rd party SDK in their programming language of choice that gives them nice language bindings, and feels like their language of choice. GraphQL is great when you are going to think about your interaction with the service in a really low level way, and ask only for the minimal data you need to do your job. But these SDK writers don't know what you need, so when they build their object models, they just do so by putting everything in. At which point you are pretty much back to where we started.

I think GraphQL is going to work out really well for really popular services (like github) where people are willing to take the hit to go super low level. Or where you know the backend details well enough to understand the cost differential of asking for different attributes. But I don't think it obviates trying to come up with a server side versioning for APIs in Open Source.



Exploring and discovering how things are more complicated, with a focus on climate and software