Category Archives: Climate & Energy

MQTT, Kubernetes, and CO2 in NY State

Back in November we decided to stop waiting for our Tesla Model 3 (ever changing estimates) and bought a Chevy Bolt EV (which we could do right off the lot). A week later we had a level 2 charger installed at home, and a work order in for a time of use meter. Central Hudson's current time of use peak times are just 2 - 7pm on weekdays, and everything else is considered off peak. That's very easy to not charge during, but is it actually the optimal time to charge? Especially if you are trying to limit your CO2 footprint on the electricity? How would we find out?

The NY Independent System Operator (ISO) generates between 75% and 85% of the electricity used in the state at any given time. For the electricity they generate, they provide some very detailed views about what is going on.

There is no public API for this data, but they do publish CSV files at 5 minute resolution on a public site that you can ingest. For current day they are updated every 5 to 20 minutes. So you can get a near real time view of the world. That shows a much more complicated mix of energy demand over the course of the day which isn't just about avoiding the 2 - 7pm window.

Building a public event stream

With my upcoming talk at IndexConf next week on MQTT, this actually jumped up as an interesting demonstration of that. Turn these public polling data sets into an MQTT live stream. And, add some data calculation on top to calculate what the estimated CO2 emitted per kWh is currently. The entire system is written as a set of micro services on IBM Cloud running in Kubernetes.

The services are as follows:

  • ny-power-pump - a polling system that is looking for new published content and publishing it to an MQTT bus
  • ny-power-mqtt - A mosquitto MQTT server (exposed at It can be anonymously read by anyone
  • ny-power-archive - An mqtt client that's watching the MQTT event stream and sending data to influx for time series calculations. It also exposes recent time series as additional MQTT messages.
  • ny-power-influx - influx time series database.
  • ny-power-api - serves up a sample webpage that runs an MQTT over websocket bit of javascript (available at


MQTT is a light weight message protocol using a publish / subscribe server. It's extremely popular in the Internet of Things space because of how simple the protocol is. That lets it be embedded in micro controllers like arduino.

MQTT has the advantage of being something you can just subscribe to, then take actions only when interesting information is provided. For a slow changing data stream like this, giving applications access to an open event stream means being able to start doing something more quickly. It also drastically reduces network traffic. Instead of constantly downloading and comparing CSV files, the application gets a few bytes when it's relevant.

The Demo App

That's the current instantaneous fuel mix, as well as the estimated CO2 per kWh being emitted. That's done through a set of simplifying assumptions by looking at 2016 historic data (explained here, any better assumptions would be welcomed).

The demo app also includes an MQTT console, where you can see the messages coming in that are feeding it as well.

The code for the python applications running in the services is open source here. The code for the deploying the microservices will be open sourced in the near future after some terrible hardcoding is removed (so others can more easily replicate it).

The Verdict

While NY State does have variability in fuel mix, especially depending on how the wind load happens. There is a pretty good fixed point which is "finish charging by 5am". That's when there is a ramp up in Natural Gas infrastructure to support people waking up in the morning. Completing charging before that means the grid is largely Nuclear, Hydro, and whatever Wind is available that day, with Natural Gas filling in some gaps.

Once I got that answer, I set my departure charging schedule in my Chevy Bolt. If the car had a more dynamic charge API, you could do better, and specify charging once it flat lined at 1am, or dropped below a certain threshold.

Learn more at IndexConf

On Feb 22nd I'll be diving into MQTT the protocol, and applications like this one at IndexConf in San Francisco. If you'd love to discuss more about turning public data sets into public event streams with the cloud, come check it out.

Power usage after going Geothermal and EV

In November 2017 we replaced our Fuel Oil Heating system with a Geothermal one from Dandelion and bought a Chevy Bolt EV, which we're using as the primary car in the house. That for us means about 1000 miles a month on it. Central Hudson never actually read our meter in January, so applied an estimated based on our old usage. We finally got a meter reading, so now have a 2 month power usage that I can compare to the last couple of years.

By the Numbers

4700 kWh.

That seems like a lot, but I do have counters on both the furnace and the EV, which were ~2200 kWh and ~800 kWh respectively during this time period. Which leaves us at 1700 kWh for the rest of our load. That's compares to 1600 kWh last year, and 1500 kWh the year before.

There is also new electric load in the hot water system, which seems to be running pretty efficiently getting dumped waste heat from the water furnace.

This includes the stretch of time where we had a 14 day cold snap with 20 degree below average temperatures (ending with a record low). So while it's hard to compare to last year directly, it's pretty favorable. I'm sure that were we on oil we'd have had at least one tank fill during that window if not two, the oil trucks have been running pretty constant in the neighborhood.


Opening the power bill had a momentary "oh wow". But then realizing we no longer have an oil bill, and we've only paid for 1 or 2 tanks of gas in the Subaru in this window puts the whole thing in perspective.

Getting to a Zero Carbon Grid

This talk by Jesse Jenkins at UPENN is one of the best looks at what doing deep decarbonization of the grid really looks like. Jenkins is a PhD candidate at MIT researching realistic paths to get our electricity sector down to zero carbon emissions.

Price vs. Value

He starts with the common and simple refrain we all have, which is that research investments in solar have driven down the cost below that of fossil fuels, that cross over point has happened, and renewables will just take off and take over.

But that's the wrong model. Because of the intermitency of Wind and Solar, after a certain saturation point the wholesale value of a new MWh of their energy keeps decreasing. This has already been seen in practice in energy markets with high penetration.

 Sources of Energy

The biggest challenge is not all sources of energy are the same.

Jenkins bundles these into 3 categories. Renewables are great at Fuel savings, providing us a way not to burn some fuel. We also need a certain amount of fast burst on the grid, today this is done with Natural Gas Peaker plants, but demand hydro and energy storage fit that bill as well. In both of these categories we are making good progress on new technologies.

However, in the Flexible base camp, we are not. Today that's being provided by Natural Gas and Coal plants, and some aging Nuclear that's struggling to compete with so much cheap Natural Gas on the market.

How the mix changes under different limits

He did a series of simulations about what a price optimal grid looks like under different emissions limits given current price curves.

Under a relatively high emissions threshold the most cost efficient approach is about 40% renewables on the grid, some place for storage. The rest of the power comes from natural gas. 16% of solar power ends up being curtailed during the course of the year, which means you had to overbuild solar capacity to get there.

Crank down the emissions limit and you get more solar / wind, but you get a lot of curtailment. This is a 70% renewable grid. It's also got a ton of over build to deal with the curtailment.

But if you want to take the CO2 down further, things get interesting. 

Because of the different between price and value, relatively high priced Nuclear makes a return (Nuclear is a stand in for any flexible base source, it's just the only one we current have in production that works in all 50 states). There still is a lot of overbuild on solar and wind, and huge amounts of curtailment. And if you go for basically zero carbon grid, you get something a little surprising.

Which is the share of renewables goes down. They are used more efficiently, there is less curtailment. These are "cost optimal" projections with emissions targets fixed. They represent the cheapest way to get to a goal.

The important take away is that we're in this very interesting point in our grid evolution where cheap Natural Gas is driving other zero carbon sources out of business because we aren't pricing Carbon (either through caps or direct fees). A 40 - 60% renewables grid can definitely emerge naturally in this market, but you are left with a lot of entrenched Natural Gas. Taking that last bit off the board with renewables is really expensive, which means taking that path is unlikely.

But 100% Renewables?

This is in contrast to the Mark Jacobson 100% renewables paper. Jenkins points out that there have really been two camps of study. One trying to demonstrate the technical ability to have 100% renewables, the other looking at realistic pathways to zero carbon grid. Proving that 100% renewables is technically possible is a good exercise, but it doesn't mean that it's feasible from a land management, transmission upgrade, and price of electricity option. However none of the studies looking at realistic paths landed on a 100% renewables option.

Jenkins did his simulation with the 100% renewables constraint, and this is what it looked like.

When you pull out the flexible base you end up with a requirement for a massive overbuild on solar to charge sources during the day. Much of the time you are dumping that energy because there is no place for it to go. You also require storage at a scale that we don't really know how to do.

Storage Reality Check

The Jacobson study (and others) make some assumptions about season storage of electricity of 12 - 14 weeks of storage. What does that look like? Pumped hydro is currently the largest capacity, and most efficient way to store energy. Basically you pump water behind a dam when you have extra / cheap energy, then you release it back through the hydro facility when you need it. It's really straight forward tech, and we have some on our grid already. But scale matters.

The top 10 pumped hydro facilities combined provide us 43 minutes of grid power.

One of the larger facilities is in Washington state it is a reservoir 27 miles long, you can see it from space. It provides 3 1/2 minutes grid average power demand.

Pumped hydro storage is great, where the geography supports it. But the number of those places is small, and it's hard to see their build out increasing dramatically over time.

Does it have to be Nuclear?

No. All through Jenkins presentation Nuclear was a stand in for any zero carbon flexible base power source. It's just the only one we have working at scale right now. There other other potential technologies including burning fossil fuels but with carbon capture and storage, as well as engineered geothermal.

Engineered Geothermal was something new to me. Geothermal electricity generation today is very geographically limited you need to find a place where you have a geologic hot spot, and an underground water reserve, that's turning that into steam you can run through generators. It's pretty rare in the US. Iceland gets about 25% of it's power this way, but it has pretty unique geology.

However, the fracking technology that created the natural gas boom openned a door here. You can pump water down 2 miles into the earth and artificially create conditions to produce steam and harvest it. It does come with the same increase in seismic activity that we've seen in fracking, but there are thoughts on mitigation.

It's all trade offs

I think the most important take away is there is no silver bullet in this path forward. Everything has downsides. The land use requirements for solar and wind are big. In Jenkins home state of Massachusetts in order to get to 100% renewables it would take 7% of the land area. That number seems small, until you try to find it. On the ground you can see lots of people opposing build outs in their area (I saw a Solar project for our school district get scuttled in this way).

In the North East we actually have a ton of existing zero carbon energy available in Hydro Quebec, that's trapped behind not having enough transmission capacity. Massachusetts just attempted to move forward with the Norther Pass Transmission project to replace shutting the Pilgrim Nuclear facility, but New Hampshire approval board unanimously voted against it.

Vermont's shutdown of their Yankee Nuclear plant in 2014 caused a 2.9% increase in CO2 in the New England ISO region, as the power was replaced by natural gas. That's the wrong direction for us to be headed.

The important thing about non perfect solutions is to keep as many options on the table, as long as you can. Future conditions might change in a way where some of these options become more appealing as we strive to get closer to a zero carbon grid. R&D is critical.

That makes the recent 2018 budget with increased investment credits for Carbon Capture and Storage and small scale Nuclear pretty exciting from a policy perspective. These are keeping some future doors open.

Final Thoughts


Jenkins presentation was really excellent, I really look forward to seeing more of his work in the future, and for a wider exposure on the fact that the path to a zero carbon grid is not a straight line. Techniques that get us to a 50% clean grid don't work to get us past 80%. Managing that complex transition is important, and keeping all the options on the table is critical to getting there.

No Coal this Christmas Season - Personal Climate Action you can take now

"The best time to plant a tree was 20 years ago. The second best time is now."

Over the last year we've done a lot to our house to make it much less energy intensive, bought an electric car, and got  involved in Citizens' Climate Lobby. The research for all of that created a big pile of links for me, which I've tried to summarize here, to really show how many different ways you can make an impact.

This list is customized for New York, because that's where I live, and where I've done all my research. It would be great to see other folks build local guides for their areas as well, and I'd love to link to them.

Where Energy is Used

How do we use energy in the US? Because we measure electricity in kWh, gasoline / fuel oil in gallons, natural gas in cubic feet, sizing them all up and comparing them is hard. And we don't think about them as a single energy system. At a national level our energy is used by [1]:

  • Buildings - 40%
    • Residential - 20%
    • Commercial - 20%
  • Transportation - 28%
    • Cars, Light Trucks, Motorcycles - 16.2%
    • Other Trucks - 6.4%
    • Planes - 2.2%
  • Industry - 32%
    • Petroleum Refining - 10%
    • Chemical Production - 8.6%
    • Paper Production - 3.5%
    • Metals Production - 3%

The bits of this that always surprise me is that buildings are our key use of energy. Buildings are long term infrastructure. Our house was built in 1960, there are plenty of houses in our area build in 1900. Improving existing buildings is critical to making our infrastructure more efficient. Every improvement we've made over the last couple of years will live on beyond us in this home.

The other thing that sticks out is that we use 10% of our energy budget in the US refining petroleum. Much of that to be burned in other parts of the system. Every time we prevent a gallon of gas from burning, we don't only save it's emissions, but the emissions that happened when it was refined.



Average Home Energy use in NY State

Get a home energy inspection

In NY, the NY State Energy Research and Development Agency has many programs to increase energy efficiency. One of the programs is subsidized home energy audits to give you a targeted plan about what the biggest impacts for saving energy in your home will be.

Air sealing and Attic Insulation

Our home was built in 1960, and insulated to the standard at the time (which was not much). A year ago we went forward on our energy audit recommendations and got our attic air sealed, and 8" of cellulose insulation put on top. The results were dramatic. Heating dropped about 15%, my home office (which is the far end of the HVAC), no longer needed a space heater, and summer cooling was also dramatically reduced.

Get your energy inspection first, but realize that proper insulation in your home will dramatically, and immediately change the comfort level, and your energy use.

Replace your Oil Furnace with Geothermal

About 50% of homes in NY State heat with Fuel Oil. It is one of the dirtiest way to heat your home.

If you live in the Hudson Valley or Albany regions, Dandelion is a new geothermal company offering package deals to replace your existing oil system with a ground source heat pump. They put a well or two in your front yard, put a sealed tube down it, then use the 50 degree earth and a compressor to heat your home. Heat pumps get about 4 units of heat for every unit of electricity they consume. Ours has been in for about a month, and so far we're in love. So much quieter, no whiffs of oil smoke, and much more even distribution of heat in the house (it runs the fan slower and longer).

When I did the math, this was the single biggest climate impact we could make. This takes 700 gallons of fuel oil off the table. In comparison, we used about 500 gallons of gasoline in an average year between our cars.

Replace your Oil Furnace with... anything else

Seriously, Fuel Oil is terrible for the environment. While Natural Gas and Propane are still fossil fuels, they emit a lot less both in creating them, and when they burn. If you can't go the full hog to something like a heat pump, changing from Oil to NG or Propane will reduce your emissions on heating to about 1/2 of what they were before.


If you've not yet replaced all your lighting in your house with LEDs, do that now. They only use about 20% of the electricity of incandescent bulbs, are more efficient than even fluorescent, and last an incredibly long time (25 year lifespans are common).

If you are a Central Hudson customer you can get 60W replacement bulbs for $1 each. Just do it. While lighting use is overall a pretty low part of your energy budget, it is also very actionable if you haven't done the conversion to LEDs yet. And, LED lights fit in christmas stockings.


The path to decarbonizing the economy is to electrify everything, while simultaneously making the electric grid less carbon intensive.


NY State's energy production is relatively low carbon, but if we are going to fully decarbonize we do need to reduce natural gas consumed for electricity as much as possible.

Choose a Green ESCO

NY State allows you to choose your energy producer (energy services company, or ESCO). There are a number of companies that provide you with energy from wind farms that they are building regionally. This typically mean a small rise in your energy costs, but that comes with supporting the build out of new renewables.

Two good options in our area are:

Community Solar

NY State has new rules in place that allow for Community Solar in our area. These are small scale (2 Mega Watt or less) facilities that you can sign up and get your power from solar even if you can't put solar panels on your roof (you have bad site, or are a renter).

Solarize Hudson Valley has sign up information for folks in the area. If you are in the Central Hudson power generation region, Nexamp is building a facility in Wappingers Falls. We've signed up, and starting in May of 2018, will be getting our power from solar.

Carbon Offsets

If there is nothing on the list that works for you, but you still want to have an impact on reducing your carbon footprint, consider some kind of carbon offset. Carbon offset projects work to capture carbon, or reduce emissions from something like a landfill. We all share one atmosphere, so any way you reduce emissions helps.

The carbon offset market is a wildly confusing place as an individual. As a NY (or North East) resident, the Carbon Reduction Certificates from the Adirondack Counsil is great. Each certificate is used to buy 1 ton of CO2 off the Regional Greenhouse Gas Initiative annual carbon auction (a carbon trading system for power companies that 9 states have agreed to, and NJ and VA might be joining soon). The price for a ton of carbon on the RGGI is still pretty low, so left over proceeds go to their micro grants program which support local energy efficiency and emissions reduction.


Get Engaged

Right now, you need to do something extra, or out of the ordinary to have an impact on climate change. Citizens' Climate Lobby is a political action group looking to change that, by pricing carbon in the economy. A real price on carbon would make doing the efficient thing, also be doing the cheaper thing, which would make it the default choice in most situations.

We've got a local chapter that meets in Beacon, NY once a month, and so if you want to flex your political muscles, as well as your economic ones, sign up and join us.

It all maters

Every action you make matters. And the exciting fact is that there are so many things you can do now to have an impact (including many things not on this list). So take a minute this holiday season and think about how you can take a little bit of coal out of your own Christmas season.


Much warmer summers

The map above, based on a new analysis from the Climate Impact Lab, shows how 95-degree days (35 degrees Celsius) are expected to multiply this century if countries take moderate climate action. In this scenario, countries would take some measures, but not drastic ones, to curb emissions — roughly the trajectory of the current pledges under the Paris climate agreement.

The resulting global warming would still cause significant shifts for many cities. In Washington, from 1986 to 2005, an average of seven days each year had temperatures of at least 95 degrees. By the end of the century, the city can expect 29 of these extremely hot days per year, on average. (The likely range is 14 to 46 hot days per year.)

Source: 95-Degree Days: How Extreme Heat Could Spread Across the World - The New York Times

Good analysis on what the impacts of the hotter days coming are going to be. 95 F is as reasonable an arbitrary measuring point as anything else, we're approaching body temperature there. The article looks at the world under the Paris agreement, as well as without it. The differences are striking.

Interestingly, commercial crop yields (specifically corn and soybeans) start to drop after 84 F. I hadn't realized that, but it makes sense. That's how hotter days, even without drought, have negative impacts on our food supply.