Tag Archives: python

Python functions on OpenWhisk

Part of the wonderful time I had at North Bay Python was also getting to represent IBM on stage for a few minutes as part of our sponsorship of the conference. The thing I showed during those few minutes was writing some Python functions running in OpenWhisk on IBM's Cloud Functions service.

A little bit about OpenWhisk

OpenWhisk is an Apache Foundation open source project to build a serverless / function as a service environment. It uses Docker containers as the foundation, spinning up either predefined or custom named containers, running to completion, then exiting. It was started before Kubernetes, so has it's own Docker orchestration built in.

In addition to just the run time, it also has pretty solid logging and interactive editing through the webui. This becomes critical when you do anything that's more than trivial with cloud functions, because the execution environment looks very different than just your laptop.

What are Cloud Functions good for?

Cloud Functions are really good when you have code that you want to run after some event has occurred, and you don't want to maintain a daemon sitting around polling or waiting for that event. A good concrete instance of this is Github Webhooks.

If you have a repository that you'd like to do some things automatically on a new issue or PR, doing with with Cloud Functions means you don't need to maintain a full system just to run a small bit of code on these events.

They can also be used kind of like a web cron, so that you don't need a full vm running if there is just something you want to fire off once a week to do 30 seconds of work.

Github Helpers

I wrote a few example uses of this for my open source work. Because my default mode for writing source code is open source, I have quite a few open source repositories on Github. They are all under very low levels of maintenance. That's a thing I know, but others don't. So instead of having PR requests just sit in the void for a month I thought it would be nice to auto respond to folks (especially new folks) the state of the world.

Pretty basic, it responses back within a second or two of folks posting to an issue telling them what's up. While you can do a light weight version of this with templates in github native, using a cloud functions platform lets you be more specific to individuals based on their previous contribution rates. You can also see how you might extend it to do different things based on the content of the PR itself.

Using a Custom Docker Image

IBM's Cloud Functions provides a set of docker images for different programming languages (Javascript, Java, Go, Python2, Python3). In my case I needed more content then was available in the Python3 base image.

The entire system runs on Docker images, so extending those is straight forward. Here is the Dockerfile I used to do that:

This builds with the base, and installs 2 additional python libraries: pygithub to make github api access (especially paging) easier, and a utility library I put up on github to keep from repeating code to interact with the openwhisk environment.

When you create your actions in Cloud Functions, you just have to specify the docker image instead of language environment.

Weekly Emails

My spare time open source work mostly ends up falling between the hours of 6 - 8am on Saturdays and Sundays, which I'm awake before the rest of the family. One of the biggest problems is figuring out what I should look at then, because if I spend and hour figuring that out, then there isn't much time to do much that requires code. So I set up 2 weekly emails to myself using Cloud Functions.

The first email looks at all the projects I own, and provides a list of all the open issues & PRs for them. These are issues coming in from other folks, that I should probably respond to, or make some progress on. Even just tackling one a week would get me to a zero issue space by the middle of spring. That's one of my 2018 goals.

The second does a keyword search on Home Assistant's issue tracker for components I wrote, or that I run in my house that I'm pretty familiar with. Those are issues that I can probably meaningfully contribute to. Home Assistant is a big enough project now, that as a part time contributor, finding a narrower slice is important to getting anything done.

Those show up at 5am in my Inbox on Saturday, so it will be the top of my email when I wake up, and a good reminder to have a look.

The Unknown Unknowns

This had been my first dive down the function as a service rabbit hole, and it was a very educational one. The biggest challenge I had was getting into a workflow of iterative development. The execution environment here is pretty specialized, including a bunch of environmental setup.

I did not realize how truly valuable a robust Web IDE and detailed log server is in these environments. Being someone that would typically just run a vm and put some code under cron, or run a daemon, you get to keep all your normal tools. But the trade off of getting rid of a server that you need to keep patched is worth it some times. I think that as we see a lot of new entrants into the function-as-a-service space, that is going to be what makes or breaks them: how good their tooling is for interactive debug and iterative development.

Replicate and Extend

I've got a pretty detailed write up in the README for how all this works, and how you would replicate this yourself. Pull requests are welcomed, and discussions of related things you might be doing are as well.

This is code that I'll continue to run to make my github experience better. The pricing on IBM's Cloud Functions means that this kind of basic usage works fine at the free tier.

Syncing Sieve Rules in Fastmail, the hard way

I've been hosting my email over at Fastmail for years, and for the most part the service is great. The company understands privacy, contributes back to open source, and is incredibly reliable. One of the main reasons I moved off of gmail was their mail filtering system was not fine grained enough to deal with my email stream (especially open source project emails). Fastmail supports sieve, which lets you write quite complex filtering rules. There was only one problem, syncing those rules.

My sieve rules are currently just north of 700 lines. Anything that complex is something that I like to manage in git, so that if I mess something up, it's easy to revert to known good state.

No API for Sieve

Fastmail does not support any kind of API for syncing Sieve rules. There is an official standard for this, called MANAGESIEVE, but the technology stack Fastmail uses doesn't support it. I've filed tickets over the years that mostly got filed away as future features.

When I first joined Fastmail, their website was entirely classic html forms. Being no slouch, I had a python mechanize script that would log in as me, then navigate to the upload form, and submit it. This worked well for years. I had a workflow where I'd make a sieve change, sync via script, see that it generated no errors, then commit. I have 77 commits to my sieve rules repository going back to 2013.

But, a couple of years ago the Fastmail team refreshed their user interface to a Javascript based UI (called Overture). It's a much nicer UI, but it means it only works with a javascript enabled browser. Getting to the form box where I can upload my sieve rules is about 6 clicks. I stopped really tweaking the rules regularly because of the friction of updating them through clear / copy / paste.

Using Selenium for unintended purposes

Selenium is pretty amazing web test tool. It gives you an API to drive a web browser remotely. With recent versions of Chrome, there is even a headless chrome driver, so you can do this without popping up a graphics window. You can drive this all from python (or your language of choice).

An off hand comment by Nibz about using Selenium for something no one intended got me thinking: could I manage to get this to do my synchronization?

Answer, yes. Also, this is one of the goofiest bits of code that I've ever written.

Basic Flow

I won't do a line by line explanation, but there are a few concepts that make the whole thing fall in line.

The first is the use of WebDriverWait. This is an OvertureJS application, which means that clicking parts of the screen trigger an ajax interaction, and it may be some time before the screen "repaints". This could be a new page, a change to the existing page, an element becoming visible. Find a thing, click a thing, wait for the next thing. There is a 5 click interaction before I get to the sieve edit form, then a save button click to finish it off.

Finding things is important, and sometimes hard. Being an OvertureJS application, div ids are pretty much useless. So I stared a lot in Chrome inspector at what looked like stable classes to find the right things to click on. All of those could change with new versions of the UI, so this is fragile at best. Some times you just have to count, like finding the last textarea on the Rules page. Some times you have to inspect elements, like looking through all the buttons on a page to find the one that says "Save".

Filling out forms is done with sendKeys, which approximates typing by sending 1 character every few milliseconds. If you run non headless it makes for amusing animation. My sieve file is close to 20,000 characters, so this takes more than a full minute to put that content in one character at a time. But at least it's a machine, so no typos.

The Good and the Bad

The good thing is this all seems to work, pretty reliably. I've been running it for the last week and all my changes are getting saved correctly.

The bad things are you can't have 2 factor enabled and use this, because unlike things like IMAP where you can provision an App password for Fastmail, this is really logging in and pretending to be you clicking through the website and typing. There are no limited users for that.

It's also slow. A full run takes

It's definitely fragile, I'm sure an update to their site is going to break it. And then I'll be in Chrome inspector again to figure out how to make this work.

But, on the upside, this let me learn a more general purpose set of tools for crawling and automating the modern web (which requires javascript). I've used this technique for a few sites now, and it's a good technique to add to your bag of tricks.

The Future

Right now this script is in the same repo as my rules. This also requires setting up the selenium environment and headless chrome, which I've not really documented. I will take some time to split this out on github so others could use it.

I would love it if Fastmail would support MANAGESIEVE, or have an HTTP API to fetch / store sieve rules. Anything where I could use a limited app user instead of my full user. I really want to delete this code and never speak of it again, but a couple of years and closed support tickets later, and this is the best I've got.

If you know someone in Fastmail engineering and can ask them about having a supported path to programatically update sieve rules, that would be wonderful. I know a number of software developers that have considered the switch to Fastmail, but stopped when the discovered that updating sieve can only be done in the webui.

Updated (12/15/2017): via Twitter the Fastmail team corrected me that it's not Angular, but their own JS toolkit called OvertureJS. The article has been corrected to reflect that.

 

Notes from North Bay Python

North Bay Python marqee at the Mystic Theatre in Petaluma, CA

I had the pleasure of attending the first North Bay Python conference in Petaluma, CA this past weekend. IBM was a sponsor, and I gave a few quick remarks about doing python serverless actions on OpenWhisk. My two days there were full of wonderful talks and interactions with a slice of the python community.

One of the reasons I love low cost (to attendee) regional conferences like North Bay Python is that it makes technology conferences more accessible. For 40% of the 250 attendees, this was the first technology conference they'd ever gone to. Not everyone lives in New York City or San Francisco (or wants to), and having local events is critical to expanding the range of voices in the technology community.

There were tons of great talks, you can watch them all here. But I'll highlight a few moments that I'll keep with me for a while.

Fortran on stage

For a single track Python conference, we actually got to see FORTRAN in 2 different talks. It's probably more FORTRAN than I've ever read before.

Catherine Moroney is part of the team that does analysis of satellite images from the LandSat program. They've got a lot of optimized FORTRAN and C code for processing these images. But FORTAN and C aren't great languages for writing new features to orchestrate these lower level transforms. She uses Python to do this work, and can seamlessly pass data back and forth from Python to FORTRAN for data crunching. It was great to see how and when a hybrid approach like this makes the developers much more effective.

Christopher Swenson tackled FORTRAN from the other side. He hacked together a FORTAN IV interpretter in Python, so that he could run Colossal Cave Adventure (originally written for the PDP-11) as a text message game using the Twilio API. His talk wandered through some of the interesting quirks of now extinct programming languages, and the systems they were written for. This is a world before ASCII as we know it became a standard, and the idea of 32bit integers really hadn't emerged. 36bit integers were used to store 5, 7bit characters, which were later assembled into text streams.

Through the whole thing he showed snippets of FORTRAN that he had to make guesses about what it really meant, as well as be really frank on shortcuts he made to get things to work. There is no more FORTRAN IV code in the world, this didn't have to be a perfect emulator, it just had to run this one single FORTRAN IV program well enough to connect it to the internet.

You can play this by texting to +1 (669) 238-3683 right now if you want to see it in action.

Twitter Bots

Tweet: "What is Machine Learning? Easy! Machine Learning is how you automate your biases so you can think about them even less."

My vote for most hilarious talk was Benno Rice's dive into writing twitter bots. He started with pretty easy template base bots, like one producing plausible plot lines for Mid Summer Murders. This is just a stack of well crafted arrays and a random number generator.

Things got more interesting when he started diving into Markov Chain bots. Especially where you take content from a bunch of different sources. It's really easy for that to just become word salad at worst, or just confusing and "meh". He found you had to keep playing with the content mix as well as the overlap parameters to get the bots to generate something that's amusing at least some of the time. The bots he's got he doesn't let post directly, content is generated offline, and he pushes the good ones after manual review.

Benno also took a dive down the path trying to do machine learning to make these better, but mostly got worse results in his experiments. But, the story of that not working out was funny all by itself. The real lesson here is that playfulness is useful in learning some new things, and that Twitter bots are a lot of fun to build.

Search First Documentation

My vote for most takeaways that I'll personally use goes to Heidi Waterhouse for "Search-First Writing for Developers". Recently there as a mass migration of OpenStack Documentation from a dedicated docs team to all the development teams.

The heart of her message is that to any first approximation, no one reads your documentation. Users end up at your documentation when they have a problem with your software, so they are showing up a) grumpy, and b) through whatever Google terms they could guess for their problem. They are not coming through your carefully curated table of contents, they are coming from Google, and then they are skimming to find their answer. The won't follow links to other pages, this is where they are.

What that means is you need to treat every page as the only page that the user will ever see, you need to optimize your content for skimming, and you need to answer problems people actually have, not the ones you think they might have. Getting real analytics on how folks are reading your docs, and the search terms they are coming in with, is an important part of this.

Hearing all these harsh and practical words from someone that spent 15 years as a technical content author was really enlightening. I'll definitely have to watch this talk again and digest more of Heidi's lessons.

Safe Spaces

Reporting guidelines for Safety Incidents

One of the welcome trends that I've seen at tech conferences over the last 5 years is a real focus on a strict Code of Conduct, clear reporting guidelines, and making sure that folks feel safe at these events. North Bay Python did a great job on that front, and that commitment definitely was reflected in a pretty diverse speaker lineup and attendee base.

The effort they went to was highlighted further by Seán Hanson's talk on Quiet Developers. We've long known that while diversity in Tech is much lower than national averages, it's ever worse in Open Source Software. A big reason for this is members of traditionally marginalized communities really don't feel safe in these environments, or may not have the spare time to devote outside of their normal day jobs. It doesn't mean they aren't great developers, it's just that current systems are optimized for loudness as much as talent. Seán's whole talk was ways to engage and get the most out of your quiet developers, and give them what they need to really succeed. While I did need to leave about the time this talk started, I stuck around and watched from the balcony. His message was really powerful and really important to how we all evolve the tech community going forward.

Double A Plus, Would Come Again

North Bay Python was definitely worth the trip. It had a few normal quirks of a first time conference on scheduling. Being Petaluma, the Theatre didn't actually have heat, so the first few hours the first day were a bit cold in there. But it warmed up pretty quickly with 250 bodies. The biggest issue in my mind was there wasn't much common space outside of the theatre, so a hallway track wasn't really a thing. It would have been nice to have a bit more milling about time to get to know folks there, and ask follow up questions of speakers.

But all in all a great time. Looking forward to seeing how they do next year.

 

Getting Chevy Bolt Charge Data with Python

Filed under: kind of insane code, be careful about doing this at home.

Recently we went electric, and got a Chevy Bolt to replace our 12 year old Toyota Prius (who has and continues to be a workhorse). I had a spot in line for a Tesla Model 3, but due to many factors, we decided to go test drive and ultimately purchase the Bolt. It's a week in and so far so good.

One of the things GM does far worse than Tesla, is make its data available to owners. There is quite a lot of telemetry captured by the Bolt, through OnStar, which you can see by logging into their website or app. But, no API (or at least no clear path to get access to the API).

However, it's the 21st century. That means we can do ridiculous things with software, like use python to start a full web browser, log into their web application, and scrape out data..... so I did that.

The Code

This uses selenium, which is a tool used to test websites automatically. To get started you have to install selenium python drivers, as well as the chrome web driver. I'll leave those as an exercise to the reader.

After that, the process looks a little like one might expect. Start with the login screen, find the fields for user/password, send_keys (which literally acts like typing), and submit.

The My Chevrolet site is an Angular JS site, which seems to have no stateful caching of the telemetry data for the car. Instead, once you log in you are presented with an overview of your car, and it makes an async call through the OnStar network back to your car to get its data. That includes charge level, charge state, estimated range. The OnStar network is a CDMA network, proprietary protocol, and ends up taking at least 60 seconds to return that call.

This means that you can't just pull data out of the page once you've logged in, because the data isn't there, there is a spinner instead. Selenium provides you a WebDriverWait class for that, which will wait until an element shows up in the DOM. We can just wait for the status-box to arrive. Then dump its text.

The output from this script looks like this:

Which was enough for what I was hoping to return.

The Future

Honestly, I really didn't want to write any of this code. I really would rather get access to the GM API and do this the right way. Ideally I'd really like to make the Chevy Bolt in Home Assistant as easy as using a Tesla. With chrome inspector, I can see that the inner call is actually returning a very nice json structure back to the angular app. I've sent an email to the GM developer program to try to get real access, thus far, black hole.

Lots of Caveats on this code. That OnStar link and the My Chevrolet site are sometimes flakey, don't know why, so running something like this on a busy loop probably is not a thing you want to do. For about 2 hours last night I just got "there is no OnStar account associated with this vehicle", which then magically went away. I'd honestly probably not run it more than hourly. I made no claims about the integrity of things like this.

Once you see the thing working, it can be run headless by uncommenting line 18. Then it could be run on any Linux system, even one without graphics.

Again, this is one of the more rediculous pieces of code I've ever written. It is definitely a "currently seems to work for me" state, and don't expect it be robust. I make no claims about whether or not it might damage anything in the process, though if logging into a website damages your car, GM has bigger issues.

 

Python Design Patterns

A friend pointed me to this talk by Brandon Rhodes on python design patterns from PyOhio a couple of years ago.

The talk asks an interesting question: why aren't design patterns seen and talked about in the Python community. He walks through the patterns in Design Patterns: Elements of Reusable Object-Oriented Software one by one, and points out some that are features of the language, some that are used in the standard library, and some that are really applicable. All with some nice small code examples.

The thing that got me thinking though was a comment he makes both at the beginning and end of the talk. The reason you don't see these patterns in Python is because Python developers tend not to write the kind of software where they are needed. They focus on small tools that connect other components, or live within a framework.

I'm a newcomer to the community, been doing Python full time for only a few years on OpenStack. So I can't be sure whether or not it's true. However, I know there are times when I'm surprised by things that I would have expected to be solved already in the language, or incompatibilities that didn't need to be there in the python 2 to 3 transition, and wonder if these come from this community not having a ton of experience with software at large code base size, as well as long duration code bases, and the kinds of deprecation and upgrade guarantees needed there.

IPython Notebook Experiments

A week of vacation at home means some organizing, physical and logical, some spending times with friends, and some just letting your brain wander on things it wants to. One of the problems that I've been scratching my head over is having a sane basis for doing data analysis for elastic recheck, our tooling for automatically categorizing races in the OpenStack code base.

Right before I went on vacation I discovered Pandas, the python data analysis library. The online docs are quite good. However on something this dense having a great O'Reilly Book is even better. It has a bunch of really good examples working with public data sets, like census naming data. It also has very detailed background on the iPython data notebook framework, which is used for the whole book, and is frankly quite amazing. It brought back the best of my physics days using Mathematica.

screenshot_109

With the notebook server iPython isn't just a better interactive python shell. It's also a powerful webui, including python autocomplete. There is even good emacs integration, which includes supporting the inline graphing toolkit. Anything that's created in a cell will be available to future cells, and cells are individually executable. Looking at the example above, I'm setting up the basic json return from elastic search, which I only need to do once after starting the notebook.

screenshot_110

Pandas is all about data series. It's really a mere mortals interface on top of numpy, with a bunch of statistics and timeseries convenience functions added in. You'll find yourself doing data transforms a lot in it. Like my college physics professors used to say, all problems are trivial in the right coordinate space. Getting there is the hard part.

With the elastic search data, a bit of massaging is needed to get the list of dictionaries that is easily convertable into a Pandas data set. In order to do interesting time series things I also needed to create a new column that was a datetime convert of @timestamp, and pivot it out into an index.

You also get a good glimpse of the output facilities. By default the last line of an In[] block is output to the screen. There is a nice convenience method called head() to give you a summary view (useful for sanity checking). Also, this data actually has about 20 columns, so before sanity checking I sliced it down to 2 relevant ones just to make the output easier to grok.

screenshot_111

It took a couple of days to get this far. Again, this is about data transforms, and figuring out how to get from point a to point z. That might include include building and index, doing a transform on it (to reduce the resolution to day level), then resetting the index, building some computed columns, rolling everything back up in groupby clauses to compute the total number of successes and runs for each job on a certain day, and doing another computed column in this format. Here I'm also only slicing out only the jobs that didn't have a 100% success rate.

screenshot_112

And nothing would be quite complete without being able to inline visualize data. This is the same graphs that John Dickinson was creating from graphite, except on day resolution. The data here is coming from Elastic Search so we do miss a class of failures where the console logs never make it. That difference should be small at this point.

Overall this has been a pretty fruitful experiment. Once I'm back in the saddle I'll be porting a bunch of these ideas back into Elastic Recheck itself. I actually think this will make asking the interesting follow on questions on "why does a particular job fail 40% of the time?" because we can compare it to known ER bugs, as well as figure out what our unclassified percentages look like.

For anyone that wants to play, here is the iPython Notebook raw.

Unity and Pidgin

One of the things that happened once getting to Ubuntu 12.04 was that gnome-do started acting up on me. Given that it's a very minimally maintained project, I decided it was time to move on. Ubuntu's dash provides a lot of the same functionality, so I finally started using it. But, I missed a few things.

Gnome-do isn't just a launcher for programs, it's an actions engine. It has support for Pidgin buddy lists, an I even wrote an NX launcher for it. I didn't really want to give either of those up, so I started trying to figure out how to add those to Unity's launcher itself.

Unity lenses, the plugins that support results, are written in either vala or python. Given that I'm trying to reflex my python muscles now, I decided that was my way in. After a few false starts I found the One Hundred Scopes project, which is an attempt to build a whole set of Unity lenses to add functions and examples for the world.

The wikipedia example is a good starting point. It gives you an idea of how to build a custom search and return results. That enabled me to build a basic launcher that fed up file urls for launching nx sessions, and I created an pushed unity-lens-nx that implements that.

But, pidgin is a little harder. There is no file to open for pidgin, this is about communicating with another program over dbus, and catching the action to do something else with dbus. Fortuntately through the help of David Callé I figured it out. Also, once I had it I found some good tricks (and a couple of undocumented dbus methods) here -  https://github.com/gregorl/Unity-Pidgin-Lens which I used as inspiration.

The net result is unity-lens-pidgin.

2 key MHVLUG people online right now

Super + b and search you buddy list. It only displays currently online buddies, and available buddies are preferred over unavailable ones. You can get it via ppa here.

There is plenty more to do. The search results should be smarter, especially taking into account most recently contacted buddies, which means integrating with zeitgeist or something equivalent. Ideas floating around there. I'd like to do some overlays with status icons, just to give a visual clue on either protocol or current status state.

If anyone else wants to help, or has their own ideas, I'd encourage you to join in on the conversation. The One Hundred Scopes community is pretty cool, and I'm happy to make their vision a little closer to reality.