Category Archives: OpenStack

Processing OpenStack GPG keys in Thunderbird

If you were part of the OpenStack keysigning party from the summit, you are currently probably getting a bunch of emails sent by caff. This is an easy way to let a key signer send you your signed key.

These are really easy to process if you are using Thunderbird + Enigmail as your signed/encrypted mail platform. Just open up the mail attachments, right click, and import key:

screenshot_161

Once you’ve done this you’ll have included the signature in your local database. Then from the command line you can:

gpg --send-key YOURKEYID

And then you are done.

Happy GPGing!

OpenStack Summit Preview: Elastic Recheck

With OpenStack summit only a few days away, I’ve been preparing materials for my Elastic Recheck talk. Elastic Recheck is a system that we built over the last 7 months to help us data mine failures in test results in the OpenStack test system to find patterns.

The Problem

OpenStack is a complicated system, with lots of components working in an asynchronous way. This means that small timing changes can often expose some interesting issues. This is especially true in an environment like the upstream gate where we are running tests in parallel.

A good example of this is a currently open bug. If you run a security group list against all tenants at the same time someone is deleting a security group, the listing returns a 404. This is because of a nesting behavior in the list, which includes running a db get over all the items in the list to get additional details. There is an exposure window there where a security group is in the list, it’s deleted by another user, then we go back to get it, and it fails. That failure currently propagates a set of exceptions which become a 404 to the end user. Which is totally unexpected.

That window seems really small, right? Like it never could actually happen. Well, in the gate, even with only 2 – 4 API calls happening at a time, we see this 7 times a day:

Security Group Listing

The Solution

Starting during the Havana RC phase, we started turning this into a search problem. Using logstash and elastic search on the back end, we find fingerprints for known bugs. These fingerprints are queries that will give us back only test runs which seem to have failed on this particular bug.

The system includes real time reporting to Gerrit and IRC when we detect that a job failed with a known issue, and bulk reporting every 30 minutes to let us understand trends and classification rates.

Elastic Recheck Flow

Overall this has been a huge boon towards really identifying some of the key issues we expose during normal testing. What’s also been really interesting is having a system like this impacts the way that people write core project code, so that errors are more uniquely discoverable. Which is a win not only for our detection, but for debugging OpenStack in a production environment.

Learning More

If you are going to be in Atlanta, and would like to know more, you’ll have lots of opportunities.

My summit talk, which is going to be overview intended for people that want to learn more about the project and technique.

Elastic Recheck – Tools for Finding Race Conditions in OpenStack
Date: Thursday, May 15th
Time: 2:20pm
Room: B206
Track: Related Open Source Projects

We’ll also be doing a design summit session for people that are interested in contributing to the project, and helping us set priorities for the next cycle. Wed, 9:50am in the Infrastructure Track.

Also, feel free to find me anywhere to chat about Elastic Recheck. I’m always happy to talk about it, especially if you are interested in getting involved in the effort.

I believe the summit talk will be recorded, and I’ll post links to the video once it’s online for people that can’t make it to Atlanta.

Helpful Gerrit Queries (Gerrit 2.8 edition)

Gerrit got a very nice upgrade recently which brings in a whole new host of features that are really interesting. Here are some of the things you should know to make use of these new features. You might want to read up on the basics of gerrit searches here: Gerrit queries to avoid review overload, before getting started.

Labels

Gone are the days of -CodeReview-1, we now have a more generic mechanism called labels. Labels are a lot more powerful because they can specify both ranges as well as specific users!

For instance, to select everything without negative code reviews:

status:open NOT label:Code-Review<=-1

Because we now have operators, we can select for a range of values, so any negative (-1, -2, or any high negative value should it get implemented in the future) matches. Also negation is done with the ‘NOT’ keyword, and notable that CodeReview becomes label:Code-Review in the new system.

Labels exist for all three columns. Verified is what CI bots vote in, and Workflow is a combination of the Work in Progress (Workflow=-1) and Approved (Workflow=1) states that we used to have.

Labels with Users

Labels get really power when you start adding users to them. Now that we have a ton of CI bots voting, with regular issues in their systems, you might want to filter out by changes that Jenkins currently has a positive vote on.

status:open label:Verified>=1,jenkins

This means that changes which do not yet have a Jenkins +1 or +2 won’t be shown in your list. Hiding patches which are currently blocked by Jenkins or it hasn’t reported on yet. If you want to see not yet voted changes, you could change that to >=0.

Labels with Self

This is where we get really fun. There is a special user, self, which means your logged in id.

status:open NOT label:Code-Review>=0,self label:Verified>=1,jenkins NOT label:Code-Review<=-1

This is a list of all changes that ‘you have not yet commented on’, that don’t have negative code reviews, and that Jenkins has passing results. That means this query becomes a todo list, because as you comment on changes, positive, negative, or otherwise, they drop out of this query.

If you also drop all the work in progress patches:

status:open NOT label:Code-Review>=0,self label:Verified>=1,jenkins NOT label:Code-Review<=-1 
  NOT label:Workflow<=-1

then I consider this a basic “Inbox zero” review query. You can apply this to specific projects with “project:openstack/nova”, for instance. Out of this basic chunk I’ve built a bunch of local links to work through reviews.

File Matching

With this version of gerrit we get a thing called secondary indexes, which means we get some more interesting searching capabilities. which basically means we also have a search engine for certain other types of queries. This includes matching changes against files.

status:open file:^.*/db/.*/versions/.* project:^openstack.*

is a query that looks at all the outstanding changes in OpenStack that change a database migration. It’s currently showing glance, heat, nova, neutron, trove, and storyboard changes.

Very helpful if as a reviewer you want to keep an eye on a cross section of changes regardless of project.

Learning more

There are also plenty of other new parts of this query language. You can learn all the details in the gerrit documentation.

We’re also going to work at making some of these “inbox zero” queries available in the gerrit review system as a custom dashboard, making it easy to use it on any project in the system without building local bookmarks to queries.

Happy reviewing!

 

Robert Muth: Better Bash Scripting in 15 Minutes

Better Bash Scripting in 15 Minutes. The tips and tricks below originally appeared as one of Google’s “Testing on the Toilet” TOTT episodes. This is a revised and augmented version.

via Robert Muth: Better Bash Scripting in 15 Minutes.

Some good bits in here. We’ve implemented some of them in devstack, and I think a few more (like uninitialized and enforcing double brackets on all conditionals) would be helpful. It also makes me think about things to enforce in bash8.

Devstack Vagrant

Devstack is tooling for OpenStack to make it easy to bring up an OpenStack environment based on the latest git trees. It’s used extensively in upstream testing, and by many OpenStack developers to set up dev/test environments.

One of my personal challenges in working on Devstack was testing devstack itself. Relying on the upstream gate means we have a limited number of configurations, and when something goes wrong, iterating on a fix is hard. Even more importantly, the upstream gate is currently only a single node test environment.

A month ago I hacked out a new tool – devstack-vagrant (available on github).

DevstackVagrant

Devstack vagrant provides a customized Vagrant environment that will build a 2 node devstack cluster under VirtualBox. The basic model is 2 devstack nodes (a controller and a compute) that bridge through a physical interface on your host. The bridged interface is set as the default route in the nodes so that 2nd level guests created on top of this 2 node devstack can route to the outside world.

The nodes start and build from official Ubuntu 12.04 cloud images, and are customized using the puppet provisioning support in vagrant. There are a few config variables you need to set in a config.yaml, including hostnames, bridge interface, and the password hash you want your stack user to have. Basically enough to bootstrap the environment and then run devstack from upstream git.

I added a bit of additional logic to the end of the normal devstack process that includes installing an Ubuntu 12.04 and Fedora 20 cloud image in your glance, injecting the ssh public key for the stack user into the keyserver, and opening up ping and ssh in all the security groups.

I still consider this an expert tool at this point, as in, if it breaks you get to keep all the pieces. However, this has been useful to me so far, and given the pull requests I got the other day, seemingly is useful for others as well. Patches definitely welcomed. And if it looks like more folks want to contribute I’ll happily move to stackforge.

One of the things I’d love to do is sort out a functioning libvirt backend for vagrant (there are 2, they are both a little wonky) because then the 2nd level guests could use nested KVM and not be ridiculously slow.

This tool has already proved very useful to me, so hopefully it will be useful to others as well.

Why you should be reviewing more OpenStack code

“Read, read, read. Read everything—trash, classics, good and bad, and see how they do it. Just like a carpenter who works as an apprentice and studies the master. Read! You’ll absorb it.”

– William Faulkner

Icehouse 3 is upon us, and as someone that is on a bunch of core review teams, it means a steady drum beat of everyone asking how do they get core reviewers to review their code. My standard response has been “make sure you are also reviewing code”.

Why?

Understanding implicit style

While most projects use the hacking program to check for trivial style issues (wrong formatting), there are a lot of other parts of style that exist inside a project. What’s a good function look like? When is a project handling exceptions vs. doing checks up front. What does spacing inside functions look like. What “feels” like Nova code, and what feels foreign and odd.

This is like when you are invited to a party at someone’s house for the first time. You walk in the door, and the first thing you do is look to the host, and the guests, and figure out if people are wearing shoes in the house or not. And follow suit if there looks like there is a pattern. It’s about being polite and adapting to the local environment.

Because unless you read other people’s code, you’ll never understand these things. There are lots of patches I look at briefly, realize that they are in some whacky style that’s so foreign to the code at hand, that I don’t have the energy to figure out what the author means, and move on.

Taking load off review teams

As a core reviewer, I currently have about 800 patches right now that I could +2 or -2. Given the rate of code coming in, that might as well be infinite. And it grows all the time.

By reviewing code, even when you don’t have approval authority, you’ll be helping the review teams weed out patches which aren’t in any way ready to move forward. That’s a huge time savings, and one that I appreciate.

Even if it’s something as simple as making sure author’s provide good commit messages, that’s huge. Because I’ll completely skip over reviews with commit messages that I can’t understand. That’s your opportunity to sell me on why I should spend the next 30 minutes looking at your code. A good commit message, being really clear about what problem this code hits, and what this solution is, and why this implementation is the right approach, will make me dive in.

A terrible or unclear commit message will probably just make me ignore the code, because if the author didn’t care enough to explain that to me in the commit message, there are probably lots of issues in the code itself. Even if you and I had a conversation about this code last week, don’t assume I remember all of that. That was probably 50 code reviews ago for me, which means the context of that conversation has long since flushed from my brain.

If you review a bunch of code, you’ll understand how these things impact your ability to review code, and will naturally adapt how you write commits (including the message) to make life of a reviewer easier.

Seeing the bigger picture

People tend to start contributing in just one corner of OpenStack, but OpenStack is a big, interconnected project. What you do in one corner can effect the rest of the project. If you aren’t reviewing code and changes happening at other layers of the project, it’s really hard to know how your piece fits into the larger picture.

Changes can look fine in the small, but have a negative impact on the wider project. If you are proactive in reviewing code more broadly you can see some of that coming, and won’t be surprised when a core reviewer -2s you because you were going in a different direction than the rest of the project.

Becoming a better programmer

When I started on the OpenStack project 2 years ago I hadn’t done python in a real way for years. My python was very rusty. But one of the first things I did was start reviewing a bunch of code, especially by some of the top people in the project.

There are some really smart and really skilled people in the OpenStack project. There are people that have been part of the python community for 15+ years. People that live and breath in python. Just reading their code makes you realize some of what can be done with the language, and what the “pythonic” way of doing things is. Nothing is better training for becoming a better python developer than learning from these folks.

Some times you’ll find a real issue, because no one is perfect. Some times you’ll find something you don’t understand, and can leave a comment as a question, which you’ll probably get an answer to. But all of it will be learning. And you will become a better developer.

It does make a difference

I’ll be 100% honest, with 800+ reviews I should be looking at, I play favorites. People that I see contributing a lot on the review side (not just volume, but real quality reviews that save me time) are people who’s code I want to review, because they are contributing to the whole of the project, not just their little corner.

So that’s why you should review more code in OpenStack. It will really contribute to the project, make you a better developer, and through all this you’ll find your code is naturally aligning better with OpenStack and gets reviewed more often. Realize this is not an overnight fix, but a long term strategy for aligning with the community and becoming part of it.

OpenStack doesn’t need a leader, it just needs to evolve

Third, and perhaps the best argument against OpenStack needing a leader, is the open nature of the beast itself. It’s precisely because there’s no dominant leader that OpenStack remains so transparent and competitive – everyone’s contributions can be seen by everyone else, and this drives people to do even better.

Most likely, those who say that OpenStack needs a leader do so because of history – previous open-source projects like Java, Linux and Android have all had a ‘dictator’ at the helm, but that doesn’t necessarily mean it’s the best path for OpenStack.

via OpenStack doesn’t need a leader, it just needs to evolve | SiliconANGLE.

If you remember correctly, Linux’s leadership and development model was largely dismissed by pundits, until it had 15 years of success under it’s belt. Then it became gospel of how Open Source projects should run.

But everything evolves over time. It doesn’t really surprise me that the pundits see OpenStack’s leadership model as different, and immediately dismiss it. We’ve got 3.5 years under our belt. Maybe at 5 or 6 everyone will now say all Open Source projects need to run like OpenStack.

Which would of course be wrong. While there are certain common threads between different Open Source communities, every community is different. Why? Because Communities are made of real people. Real people with different passions, strengths, weaknesses, biases, loves, constraints, and moments of brilliance. This isn’t something you can model with spheroid approximations of upstream developers. Replicating another project’s leadership model might be easy, but in most cases isn’t what your community actually needs.

Are there areas for improvement? Sure. There always are. But improvement is a watch word for OpenStack, something we apply everywhere: to code, to process, to communication.

So I agree, we don’t need a single leader. And the evolution that continues in OpenStack will be a key strength, not a weakness as the project goes forward.