I’m hoping it pulls away some of the mystery of the process, and provides a more gentle on ramp to everything for new contributors. I’ll probably be giving some version of this again at future events, so feedback (here or on youtube) is appreciated.

Things were chugging along quite well until the installer was supposed to install additional packages. Then it bombed out (I blame the ancient CDRs). I was able to get it to at least install grub, and get the thing to boot back onto the network.

What I found myself with was a super minimal install. It didn’t yet have a normal sources.list, it didn’t have openssh-server, it didn’t have ssh client even, it didn’t have any of the normal even minimal server install tools. I had about 30 minutes of manual to typing to get the base apt repo in, and get me so I could ssh in from upstairs to drive the rest of the process.

Boostrapping a Puppet Master

This is the machine that’s my puppet master. I had a copy of the oldroot over in one of my software raid arrays, so the moment I got that mounted, I dumped over the /etc/puppet this machine should have, and tried to just puppet my way up the rest of the way. I’d been on a month long kick to puppetize my home infrastructure, so this was a promising direction.

It turns out puppet up from nothing is a little harder when you are the puppetmaster, and dnsserver for the network as well. So it was about another 30 minutes of manually installing what was needed to get my puppetmaster started. Once that was up, I managed to get the first puppet agent run in, and it was epic. 45 minutes chugging away pulling down all the policies I needed, applying packages and configs, all the kind of magic that prevented me from spending my whole day trying to figure out how I had this server setup before.

It also showed me where my policy had holes. I’ve got xfs filesystems now, so xfsprogs need to be in the base case. My libvirt setup didn’t actually install kvm, but in the super minimal install, that wasn’t there. I hadn’t gotten around to managing my openvpn server yet, that’s in there now.

If I was to do it again…

One thing I really need is both a puppet and puppetmaster bootstrapping script. Using puppet to manage your puppetmaster is cool and all, but there is a bit of snake eating it’s own tail to get you started that required a little more manual command slinging than I liked.

But, had I not had so much of my server policy encoded in puppet, I’d still be typing commands now to get that box up and running. So I’m sold on the whole process, even for a smallish IT environment, like a few home servers and remote guests.

Modafinil, which is marketed as Provigil in the United States, was first approved by the FDA in 1998 for the treatment of narcolepsy, but since then it’s become better known as a nootropic, a “smart drug,” especially among entrepreneurs. More recently, it has attracted traders like Borden who don’t just need a pick-me-up to get through a deadline; they need to be on, without a break, for months, even years at a time.

via Modafinil Is Wall Street’s New Drug of Choice — New York Magazine.

It reminds me a lot of A Deepness in the Sky, minus the giant spider aliens.

… love it.

There are a couple of reasons why the bubble is sure to burst. The first is just that it’s a bubble, and any chart which looks like the one at the top of this post is bound to end in tears at some point. But there’s a deeper reason, too — which is that bitcoins are an uncomfortable combination of commodity and currency. The commodity value of bitcoins is rooted in their currency value, but the more of a commodity they become, the less useful they are as a currency.

via The Bitcoin Bubble and the Future of Currency — Money & Banking — Medium.

Definitely one of the best articles I’ve read on Bitcoin. I think the idea of bitcoin as commodity actually makes a lot more sense, given that it’s an artificially constrained resource that’s “mined”.

Easy Planet 1.1 is now out. This is a straight port of UD Planet to D7. The only functionality which isn’t there is the disabling of aggregator menu items, which I’m not convinced was a good idea in the first place. It’s live on http://mhvlug.org/planet, for anyone that wants to see what it looks like.

One of the most important things to me was smooth transition from UD Planet installation. Because mhvlug.org was a D6 -> D7 migration, I still had udplanet tables and settings in my database. I wanted the experience of easyplanet to be: turn it on, all your stuff is back. I managed to get it there last night, so I’d be extremely interested in others that are moving from UD Planet -> Easy Planet to make sure it’s as seamless for them.

For more info, and to download the code, check out the project site.

Unlike a phone/tablet, this still means I have sane ssh access, and possibly able to still do code through an ssh connection to a server somewhere. We’ll see how the experiment holds up over the next month or so.

Honestly, the thing that’s going to give me the biggest headache is the touchpad, which is actually pretty good. But I’m someone that has 10+ years of muscle memory on the Thinkpad track point, and I don’t think I’ll ever get more than tolerant of touchpads.

We had a venue: the Mid Hudson Library System Auditorium. We had a date and time:  Wed the 5th of March, 6pm. We had a speaker: Michael Kaegler talking on Linux Firewalling. We had a mailing list and a website, and could clearly see there was interest in the group. But we’d never had a meeting, so this was the moment of truth.

But I was on a plane, racing to get back for the meeting. A month prior my job had changed, I’d started working on OpenHPI, and I’d picked up some standards work, and that meant a trip out of Portland for a 2 day standards meeting with a lot of Intel folks. I managed to get a flight which “should” get me back in time, but I had a plan B to have a Mike Salerno kick off the meeting in my absence. I landed at 5:15, was in my car by 5:30. The meeting was at least 35 minutes away, and there was traffic. Plan B was going to have to be good enough.

When I walked in the door I was blown away. The room was packed! There were at least 35 folks when I arrived, and over the course of the meeting it grew to 50. Wow. I was hoping that it would be more than just me and a few coworkers, but never had I imagined this. I wondered how long this experiment would run, fully imagining that after two years we’d run out of steam and interest and move on.

It’s been 10 years, and MHVLUG is still running strong, better than ever.

For everyone that was part of our first decade, making MHVLUG successful: Thank You. This is a community first, and it’s the people that make this awesome.

For everyone that hasn’t checked us out yet, our 10th Anniversary meeting is tomorrow. We’re going to be talking about: Linux where you least expect it. There will be cake, coffee, and conversation. There will be time for socializing and mixing, to become part of this dynamic community. Join us as we start decade number two.

You can generate your own one here.

Source Forge has had a pretty storied history. It started Open Source, all those years ago, then in the dot com collapse they stopped releasing source and instead tried to sell an onsite hosted solution, Source Forge Enterprise Edition. The Linux Technology Center was one of their few customers, providing an internal source forge for the rest of IBM. I had the “opportunity” to help debug some of that code for performance reasons, and discovered that a lot of Source Forge’s slowness was due to a major lack of understanding by the development team on how database indexes work. Those fixes flowed upstream.

Later, one of the key developers from Source Forge forked GForge from the last open source release. So we had Open Source “source forge” again. Then a couple years later the GForge team pulled the same stunt as Source Forge, tried to monetize, and seal off the source code.

Then git happened, and all these CVS / SVN based hosting solutions looked really quaint. A couple years later we had github, and the center of gravity of Open Source has been migrating ever since.

Source Forge’s current owner is Dice, the job search company, so the economics of keeping it Open Source are a little different. “What’s your github id?” is now a standard job interview question, so I can imagine the new Source Forge team has a pretty broad brush to just make Source Forge as good as they can.

I wish them luck.

These tests take on the order of 1 hour to run on a commit, which would make you immediately think the most code that OpenStack could merge in a day would be 24 commits. So how did Nova itself manage to merge 94 changes since Monday (not to mention all the other projects, which adds up to ~200 in 3 days)? The magic of this is Zuul, the gatekeeper.

Zuul is a queuing system for CI jobs, written and maintained by the OpenStack infrastructure team. It does many cool things, but what I want to focus on is the gate queue. When the gate queue is empty (yes it does happen some times), the job is simple: add a new commit, run the tests, and we’re off. What happens if there are already 5 jobs ahead of you in the gate? Let’s take a concrete example of nova.

Speculative Merge

By the time a commit has gotten this far, it’s already passed the test suites at least once, and has had at least 2 core contributors sign off on the change in code review. So Zuul assumes everything ahead of the change in the gate will succeed, and starts the tests immediately cherry picking this change on top everything that’s ahead of it in the queue.

That means that merge time on the gate is O(1), that is merging 10 changes takes the same time as 1 change. If the queue gets too big, we do eventually run out of devstack nodes, so the ability to run tests is not strictly constant time. On the run up to grizzly-3 both the cloud providers (HP and Rackspace) which contribute these VMs provided some extra quota to the OpenStack team to help keep things moving. So we had an elastic burst of OpenStack CI onto additional OpenStack public cloud resources, which is just fun to think about.

Speculation Can Fail

Of course, speculation can fail. Maybe change 3 doesn’t merge because something goes wrong in the tests. If that happens we then kick the change out of the queue, and then all the changes behind it have to be reset to pull change 3 out of the speculation. This is the dreaded gate reset, because when gate resets happen, all the time spent on speculative tests behind the failure is lost, and the jobs need to restart.

Speculation failures largely fall into a few core classes:

Jenkins crashes – it doesn’t happen often, but Jenkins is software too, and OpenStack CI tends to drive software really hard, so we force out edge cases everywhere.

Upstream service failures – we try to isolate ourselves from upstream failures as much as possible. Our git trees pull from our gerrit, not directly from github. Our apt repository is a Rackspace local mirror, not generically upstream. And the majority of pip python packages come from our own proxy server. But if someone adds a new python dependency, or a version of one updates and we don’t yet have it cached, we pass through to pypi for that pip install. On Tuesday pypi converted from HTTP to HTTPS, and didn’t fully grok the load implications, which broke OpenStack CI (as well as lots of other python developers) for a few hours when pypi effectively was down from load.

Transient OpenStack bugs – OpenStack is complicated software, 7 core components interacting with each other asynchronously over REST web services. Each core component being a collection of daemons that interact with each other asynchronously. Sometimes, something goes wrong. It’s a real bug, but only shows up under very specific timing and state conditions. Because OpenStack CI runs so many tests every day (OpenStack CI may be one of the largest creators of OpenStack guests in the world every day), very obscure edge and race conditions can be exposed in the system. We try to track these as recheck bugs, and are making them high priority to address. By definition they are hard to track down (they expose themselves on maybe 1 out of 1000 or fewer test runs), so the logs captured in OpenStack CI are the tools to get to the bottom of these.

Towards an Even Better Gate

In my year working on OpenStack I’ve found the unofficial motto of the project to be “always try to make everything better”. Continuous improvement is not just left to the code, and the tests, but the infrastructure as well.

We’re trying to get more urgency and eyes on the transient failures, coming up with ways to discover the patterns from the 1 in 1000 fails. After you get two or three that fail in the same way it helps triangulate the core issue. Core developers from all the projects are making these high priority items to fix.

On the upstream service failures the OpenStack infrastructure team already has proxies sitting in front of many of the services, but the pypi outage showed we probably need something even more robust to handle that upstream service outage, possibly rotating between pypi mirrors on the fall-through case, or a better proxy model. The team is already actively exploring solutions to prevent that from happening again.

As always, everyone is welcomed to come help us make everything better. Take a look at the recheck bugs and help us solve them. Join us on #openstack-infra and help with Zuul. Check out what the live Zuul queue looks like. All the code for this system is open source, and available under either the openstack, or openstack-infra github accounts. Patches are always welcome!

It’s interesting that LibreOffice seems to be doing a much better job than OpenOffice at removing technical debt. I think we’re already seeing the effect of that cultural split, and I expect that in the future this is going to get far more obvious.

I can’t wait to get the Android remote working for future presentations. Will be a lot of fun to drive my presentations that way.

I gave the folks there a glimpse of what’s behind a successful project that is able to integrate code from over 400 unique developers in 5 months time. I talked about planning, the design summits, the contribution and code review tools we use. But, as with every time I talk about OpenStack, the thing that really wows people is the testing infrastructure we’ve got. It was equally latched onto by the students and CIS staff in the room.

On every code submission we run style checks, unit tests (5000 of them in Nova now), and spin up a full OpenStack install and hit it with a nearly 700 test integration suite, before the first humans start looking at the code for manual review. It’s an incredibly empowering system, that means developers have a high bar to submit working code that doesn’t alter the behavior of the system. And it means that by the time the expert eyes do code review, the kinds of problems they are looking for are much more interesting.

Just this morning it meant I could look through a new proposed extension in gerrit and focus on some of the functional behavior, including understanding which kinds of code the test system has a harder time touching. The confidence that gives you as a reviewer that everything isn’t on the verge of breaking all the time, is enormous.

I’ve submitted a similar talk to the OpenStack summit, with a slightly different perspective of educating new developers on what the process from idea to code landing in the OpenStack tree is. Hoping that gets selected as it should be a good talk, and give me an excuse to polish some of my code flow diagrams a bit more.

The page is sufficiently stained with turmeric to realize how often it was made.

A little while later I noticed a Goggles Alert on my cell phone, it had scanned the image, and returned the following URL as a hit: http://littlechefapp.com/recipes/144571-chicken-curry-authentic#.UOGjR2JQCoM

Dead on. The future is pretty awesome some times.

About a year ago I converted over to using Lastpass, which means all my passwords for various websites are unique, and 12+ characters of randomness. Huge security improvement. However, it means every time I try to log in on the mobile web it’s a multi step process to jump over to the lastpass App enter master password, enter it again to get username and password copied, jump back over to mobile Chrome, copy paste into the input fields, and finally log in. This is in contrast to the Desktop experience of feeding my master password ever couple of hours, and it automatically detecting and logging me into sites when I visit them. The mobile browsing experience feels clunky and broken compared to the desktop.

How I wish Mobile Chrome supported extensions, but it’s not clear they are ever going to change that.

However, Mobile Firefox does.

Over Christmas break I figured out that lastpass actually works in mobile firefox, and after a little configuration started using Mobile Firefox instead of Mobile Chrome on both my Nexus 7 and S3. The overall browser seems roughly the same speed (maybe slightly slower), however the experience is much better. You get Ad Block, which turns the web back into something vaguely sane, and my browsing experience is now akin to the Desktop. Enough so that I’ll now use my Nexus 7 over my laptop for many browsing tasks.

Hopefully Google will eventually bring these features to their platform, but for now, the Firefox mobile strategy seems to be bearing some fruit, and reopening mobile browsers to innovation.

Dr Jeff Masters, who created Weather Underground, has a “year in” post on the weather of 2012. Which also tells the story in this remarkable video, showing a year in weather over the US in just 3 1/2 minutes (it slows down for some of the more dramatic events, which are kind of amazing). The Derecho that took out power in West Virgina, Virginia, and DC is by far the most amazing.

If you win the Powerball jackpot today, there are a few things you should know. After beating the 1 in 175 million odds, you have an 11 in 175 million chance of being killed in your car after collecting the winnings. If you survive that, you have a 327,250 to 175 million chance of being robbed of those winnings, and a 805,000 to 175 million chance that new mansion will go up in flames, according to Eve Waltermaurer, associate professor of sociology at State University of New York at New Paltz.

Probability is a bitch some times. (From the Poughkeepsie Journal)

Five hundred years is a nearly unimaginable gulf from today’s perspective. Five centuries ago, the Portuguese conquistadores were beginning their rampage through South America; Martin Luther was finishing his doctorate in theology and thinking about sin: the huge sequence of civil wars that racked Japan for over a century were raging: the Great Powers were still the Chinese empire and the Caliphate (although the latter was undergoing a shift in center of gravity towards Istanbul and the Ottoman empire). The great powers in Europe were Spain and Venice; the English speaking world was a few million barbarians occupying a handful of damp islands on the outer fringes of Europe. It’s more than twice the historical existence of the USA to this date. Of our social institutions, very few survive from that long ago: the Catholic Church (and various orders and sub-groups within it), the Japanese Monarchy, and so on. A handful of universities, banks, and other institutions. The half-life of a public corporation today is about 30 years: ten half-lives out — 300 years hence — we may expect only one in a million to survive.

The whole post is definitely worth your time, but I do keep coming back to that half life statement. We take it for granted some time that organizations that exist today will be there tomorrow. But the reality is there is nothing magical about organizations, it’s about the people. Things only get done because some decides to do them.

Contemplating the long view seems like an appropriate Sunday morning activities.

* personally guilty of this one

Contrast that with what happened in the other campaign, where this was clearly not a core part of what they were doing.