Python functions on OpenWhisk

Part of the wonderful time I had at North Bay Python was also getting to represent IBM on stage for a few minutes as part of our sponsorship of the conference. The thing I showed during those few minutes was writing some Python functions running in OpenWhisk on IBM’s Cloud Functions service.

A little bit about OpenWhisk

OpenWhisk is an Apache Foundation open source project to build a serverless / function as a service environment. It uses Docker containers as the foundation, spinning up either predefined or custom named containers, running to completion, then exiting. It was started before Kubernetes, so has it’s own Docker orchestration built in.

In addition to just the run time, it also has pretty solid logging and interactive editing through the webui. This becomes critical when you do anything that’s more than trivial with cloud functions, because the execution environment looks very different than just your laptop.

What are Cloud Functions good for?

Cloud Functions are really good when you have code that you want to run after some event has occurred, and you don’t want to maintain a daemon sitting around polling or waiting for that event. A good concrete instance of this is Github Webhooks.

If you have a repository that you’d like to do some things automatically on a new issue or PR, doing with with Cloud Functions means you don’t need to maintain a full system just to run a small bit of code on these events.

They can also be used kind of like a web cron, so that you don’t need a full vm running if there is just something you want to fire off once a week to do 30 seconds of work.

Github Helpers

I wrote a few example uses of this for my open source work. Because my default mode for writing source code is open source, I have quite a few open source repositories on Github. They are all under very low levels of maintenance. That’s a thing I know, but others don’t. So instead of having PR requests just sit in the void for a month I thought it would be nice to auto respond to folks (especially new folks) the state of the world.

#
#
# main() will be invoked when you Run This Action
#
# @param Cloud Functions actions accept a single parameter, which must be a JSON object.
#
# @return The output of this action, which must be a JSON object.
#
#

import github
from openwhisk import openwhisk as ow


def thank_you(params):
    p = ow.params_from_pkg(params["github_creds"])
    g = github.Github(p["accessToken"], per_page=100)

    issue = str(params["issue"]["number"])


    repo = g.get_repo(params["repository"]["full_name"])
    name = params["sender"]["login"]
    user_issues = repo.get_issues(creator=name)
    num_issues = len(list(user_issues))

    issue = repo.get_issue(params["issue"]["number"])

    if num_issues < 3:
        comment = """
I really appreciate finding out how people are using this software in
the wide world, and people taking the time to report issues when they
find them.
I only get a chance to work on this project on the weekends, so please
be patient as it takes time to get around to looking into the issues
in depth.
"""
    else:
        comment = """
Thanks very much for reporting an issue. Always excited to see
returning contributors with %d issues created . This is a spare time
project so I only tend to get around to things on the weekends. Please
be patient for me getting a chance to look into this.
""" % num_issues

    issue.create_comment(comment)


def main(params):
    action = params["action"]
    issue = str(params["issue"]["number"])
    if action == "opened":
        thank_you(params)
        return { 'message': 'Success' }
    return { 'message': 'Skipped invocation for %s' % action }

Pretty basic, it responses back within a second or two of folks posting to an issue telling them what’s up. While you can do a light weight version of this with templates in github native, using a cloud functions platform lets you be more specific to individuals based on their previous contribution rates. You can also see how you might extend it to do different things based on the content of the PR itself.

Using a Custom Docker Image

IBM’s Cloud Functions provides a set of docker images for different programming languages (Javascript, Java, Go, Python2, Python3). In my case I needed more content then was available in the Python3 base image.

The entire system runs on Docker images, so extending those is straight forward. Here is the Dockerfile I used to do that:

# Dockerfile for example whisk docker action
FROM openwhisk/python3action

# add package build dependencies
RUN apk add --no-cache git

RUN pip install pygithub

RUN pip install git+git://github.com/sdague/python-openwhisk.git

This builds with the base, and installs 2 additional python libraries: pygithub to make github api access (especially paging) easier, and a utility library I put up on github to keep from repeating code to interact with the openwhisk environment.

When you create your actions in Cloud Functions, you just have to specify the docker image instead of language environment.

Weekly Emails

My spare time open source work mostly ends up falling between the hours of 6 – 8am on Saturdays and Sundays, which I’m awake before the rest of the family. One of the biggest problems is figuring out what I should look at then, because if I spend and hour figuring that out, then there isn’t much time to do much that requires code. So I set up 2 weekly emails to myself using Cloud Functions.

The first email looks at all the projects I own, and provides a list of all the open issues & PRs for them. These are issues coming in from other folks, that I should probably respond to, or make some progress on. Even just tackling one a week would get me to a zero issue space by the middle of spring. That’s one of my 2018 goals.

The second does a keyword search on Home Assistant’s issue tracker for components I wrote, or that I run in my house that I’m pretty familiar with. Those are issues that I can probably meaningfully contribute to. Home Assistant is a big enough project now, that as a part time contributor, finding a narrower slice is important to getting anything done.

Those show up at 5am in my Inbox on Saturday, so it will be the top of my email when I wake up, and a good reminder to have a look.

The Unknown Unknowns

This had been my first dive down the function as a service rabbit hole, and it was a very educational one. The biggest challenge I had was getting into a workflow of iterative development. The execution environment here is pretty specialized, including a bunch of environmental setup.

I did not realize how truly valuable a robust Web IDE and detailed log server is in these environments. Being someone that would typically just run a vm and put some code under cron, or run a daemon, you get to keep all your normal tools. But the trade off of getting rid of a server that you need to keep patched is worth it some times. I think that as we see a lot of new entrants into the function-as-a-service space, that is going to be what makes or breaks them: how good their tooling is for interactive debug and iterative development.

Replicate and Extend

I’ve got a pretty detailed write up in the README for how all this works, and how you would replicate this yourself. Pull requests are welcomed, and discussions of related things you might be doing are as well.

This is code that I’ll continue to run to make my github experience better. The pricing on IBM’s Cloud Functions means that this kind of basic usage works fine at the free tier.

Slow AI

Charlie Stross’s keynote at the 34th Chaos Communications Congress Leipzig is entitled “Dude, you broke the Future!” and it’s an excellent, Strossian look at the future we’re barelling towards, best understood by a critical examination of the past we’ve just gone through.

Stross is very interested in what it means that today’s tech billionaires are terrified of being slaughtered by psychotic runaway AIs. Like Ted Chiang and me, Stross thinks that corporations are “slow AIs” that show what happens when we build “machines” designed to optimize for one kind of growth above all moral or ethical considerations, and that these captains of industry are projecting their fears of the businesses they nominally command onto the computers around them.

Charlie Stross’s CCC talk: the future of psychotic AIs can be read in today’s sociopathic corporations

The talk is an hour long, and really worth watching the whole thing. I especially loved the setup explaining the process of writing believable near term science fiction. Until recently, 90% of everything that would exist in 10 years already did exist, the next 9% you could extrapolate from physical laws, and only really 1% was stuff you couldn’t image. (Stross makes the point that the current ratios are more like 80 / 15 / 5, as evidenced by brexit and related upheavals, which makes his work harder).

It matches well with Clay Shirky’s premise in Here Comes Everyone, that first goal of a formal organization is future existence, even if it’s stated first goal is something else.

Do Demographics impact Leadership?

This morning on NPR there was a piece with Howard Dean about how older leaders should step aside and make way for a new generation. This has popped up in politics a bunch of times over the past couple of years, as the elected officials seem quite old. Our last presidential election gave options of a 69 and a 70 year old. With an ex president leaving office at the age of 55. The thing that most frustrates me about this reporting is it never talks about the root cause, which is demographics.

Birth Rates in the US 1943 – 2004: By Before My Ken – Wikipedia EN, Public Domain

After the baby boom, there was quite a baby bust. Births don’t tell the whole story, as immigration / emigration still move population around, people die, etc. But here is a 2015 snapshot of population by age.

Population by age in the US (2015 snapshot)

You can see that birth shape still in overall demographics, including the 1970-71 mini spike. It’s been filled in a bit by immigration, and the baby boom is less evident now that boomers are in their 60s and up and dying off. But it’s still there. And it’s important.

The Leadership Hole

About a decade ago I had the incredible opportunity to participate in a year long leadership excellence program at IBM. One of the classes was generational leadership, at a time when millennials were just about to enter the workforce. The teach talked about this demographic hole (all in generalizations).

The Baby Boomers were one of the largest generations, Gen X one of the smallest, the Millenials are nearly as big a generation as the Baby Boom. Gen X saw their parents laid off by large institutions, and the reaction is a huge upward trend in starting their own businesses. There are both less Gen Xers, and far less of them participating in large institutions still.

This means that as the Baby Boomers age out, who steps up? There aren’t actually a ton of Gen Xers there in the wings. And the Millennials are all very young and haven’t had the depth of experience yet. Do you try to fast promote them and hope they pick up enough along the way?  Many will, lots of “tribal knowledge” is going to be lost along the way. That might be good or bad depending on what habits were carried forward, but it’s unlikely to be a smooth transition regardless.

What Generational Discontinuity Looks Like

same-sex-marriage
Pew Research Center

 

FT_16.10.11_marijuanaLegal_trend
Pew Research Center

This is what generational discontinuity looks like. A flip of 60 / 40 opinion of something being bad vs. good in a 10 year period. Conventional wisdom, accepted norms, flipping really quite quickly after decades of the old attitude being around without any real changes.

Through this lens, the “why now?” of the Silence Breakers are another one of these 60 / 40 flips that we are right in the middle of the crossover. All of this was “accepted” until it wasn’t, and it will take a decade to fully entrench the new norm. Lots of people knew all this wasn’t ok, but it took Millennials standing up, and Baby Boomers dying out, to flip it.

There are Costs

It’s easy to see social progress and assume this is mostly upside. But there are costs of not carrying down experience and understanding. This analysis of the USS McCain collision showed how this leadership hole is hitting the Navy:

But the US Navy has its own particular history of creating fatigue through stress. The Navy’s Surface Warfare community has a reputation for “eating its young.” A “Zero-Defect” management mentality toward leadership has, despite entreaties from higher in the ranks, been ingrained in the surface Navy’s command structure. When something happens, junior officers get burned. There is no learning from mistakes—only punishment for making them.

Couple that with tougher tasks than ever. Over the past decade, as a Government Accountability Office study in 2015 noted, “to meet the increasing demands of combatant commanders for forward presence in recent years, the Navy has extended deployments; increased operational tempos; and shortened, eliminated, or deferred training and maintenance.” And to increase the availability of ships for ongoing missions in the Mediterranean and the Western Pacific, the Navy has “home-ported” ships overseas.

But the increased use of these forward deployed ships—which spent twice as much time at sea as similar ships based in the US—has had an impact on the training and maintenance of those ships in particular. In the Seventh Fleet, ships no longer even have a designated period for training. These days some Navy observers suggest the US may have the best equipped fleet in the world, but it no longer has the most skilled sailors driving them.

It’s the same pattern. Extending the terms of older generation, younger generation not getting enough mentorship, and critical bits breaking down when younger generation are put on the spot without enough preparation.

Patterns in our Demographics

The predictive power of this demographic hole is important. Like a warming climate it’s not going to tell you exactly which new super storm is going to hit which state. But it does tell us to expect an increase in:

  • Breakdowns in large systems / companies / organizations as the leadership hole prevented the pass down of unwritten rules and important tribal knowledge that kept the system working
  • Quick 60 / 40 opinion flips as the lack of gradual pass down of culture caused the younger generation to reevaluate their definition of normal

I just wish that more folks reporting on the events of the day pondered some of these larger factors and forcing functions of history like population demographics.

P.S. That’s just from the disruption of flow of information through population bottle necks. If you start thinking about how opportunity exists when you hit the job market if you are in a dip or peak of the population, that gets fascinating as well. Malcolm Gladwell proposed that as part of the rise of the tech giants in Outliers. While he is always a bit light on research, it’s interesting to ponder.