Bash trick of the week – call stacks

For someone that used to be very vocal about hating shell scripting, I seem to be building more and more tools related to it every day. The latest is caller (from “man bash”):

caller [expr]
Returns the context of any active subroutine call (a shell function or a script executed with the . or source builtins). Without expr, caller displays the line number and source filename of the current subroutine call. If a non-negative inte‐ ger is supplied as expr, caller displays the line number, subroutine name, and source file corresponding to that position in the current execution call stack. This extra information may be used, for example, to print a stack trace. The current frame is frame 0. The return value is 0 unless the shell is not executing a subroutine call or expr does not correspond to a valid position in the call stack.

This means that if your bash code makes heavy use of functions, you can get the call stack back out. This turns out to be really handy for things like writing testing scripts. I recently added some more unit testing to devstack-gate, and used this to make it easy to see what was going on:

# Utility function for tests
function assert_list_equal {
    local source=$(echo $1 | awk 'BEGIN{RS=",";} {print $1}' | sort -V | xargs echo)
    local target=$(echo $2 | awk 'BEGIN{RS=",";} {print $1}' | sort -V | xargs echo)
    if [[ "$target" != "$source" ]]; then
        echo -n `caller 0 | awk '{print $2}'`
        echo -e " - ERROR\n $target \n != $source"
        ERRORS=1
    else
    # simple backtrace progress detector
        echo -n `caller 0 | awk '{print $2}'`
        echo " - ok"
    fi
}

The output ends up looking like this:

ribos:~/code/openstack/devstack-gate(master)> ./test-features.sh 
test_full_master - ok
test_full_feature_ec - ok
test_neutron_master - ok
test_heat_slow_master - ok
test_grenade_new_master - ok
test_grenade_old_master - ok
test_full_havana - ok

I never thought I’d know this much bash, and I still think data structure manipulation is bash is craziness, but for imperative programming that’s largely a lot of command calls, this works pretty well.

Greed and the Wright Brothers – NYTimes.com

The Wright brothers’ critical insight was the importance of “lateral stability” — that is, wingtip-to-wingtip stability — to flight. And their great innovation was something they called “wing warping,” in which they used a series of pulleys that caused the wingtips on one side of the airplane to go up when the wingtips on the other side were pulled down. That allowed the Wrights’ airplane to make banked turns and to correct itself when it flew into a gust of wind.

But when the Wrights applied for a patent, they didn’t seek one that just covered wing warping; their patent covered any means to achieve lateral stability. There is no question what the Wrights sought: nothing less than a monopoly on the airplane business — every airplane ever manufactured, they believed, owed them a royalty. As Wilbur Wright, who was both the more domineering and the more inventive of the two brothers, put it in a letter: “It is our view that morally the world owes its almost universal system of lateral control entirely to us. It is also our opinion that legally it owes it to us.”

What was Curtiss doing in the meantime? In addition to coming up with the idea of adding wheels for easier takeoffs and landings, he invented an entirely different system for dealing with lateral stability, a system of flaps that went up and down and controlled the wings. (Airplane manufacturers today still use that basic insight.) The Wrights responded by filing a lawsuit, claiming that Curtiss was violating their patents. The litigation would consume them literally until the day Wilbur Wright died.

via Greed and the Wright Brothers – NYTimes.com.

The problem with the Intellectual Property is that it incentivizes people to sit on their laurels once they’ve captured an idea, so stiffles the next round of innovation. Most of these ideas aren’t nearly as revolutionary as you think, as even ground break ideas are typically simultaneously invented by independent groups. There is a great survey of this across multiple major inventions in human history in Steven Johnson’s Where Good Ideas Come From.

Robert Muth: Better Bash Scripting in 15 Minutes

Better Bash Scripting in 15 Minutes. The tips and tricks below originally appeared as one of Google’s “Testing on the Toilet” TOTT episodes. This is a revised and augmented version.

via Robert Muth: Better Bash Scripting in 15 Minutes.

Some good bits in here. We’ve implemented some of them in devstack, and I think a few more (like uninitialized and enforcing double brackets on all conditionals) would be helpful. It also makes me think about things to enforce in bash8.

We live on fragile hopes and dreams

OpenSSL isn’t formally verified!?

No, neither is any part of your browser, your kernel, your hardware, the image rendering libraries that your browser uses, the web servers you talk to, or basically any other part of the system you use.

The closest to formally verified in your day-to-day life that you’re going to get may well be the brakes on your car, or the control systems on a jet engine. I shit you not.

We live on fragile hopes and dreams.

via My Heart Bleeds for OpenSSL | Coder in a World of Code.

At lot of the internet is learning a lot more about how software in the wild functions after heartbleed. I found that statement to be one of the best summaries.

Devstack Vagrant

Devstack is tooling for OpenStack to make it easy to bring up an OpenStack environment based on the latest git trees. It’s used extensively in upstream testing, and by many OpenStack developers to set up dev/test environments.

One of my personal challenges in working on Devstack was testing devstack itself. Relying on the upstream gate means we have a limited number of configurations, and when something goes wrong, iterating on a fix is hard. Even more importantly, the upstream gate is currently only a single node test environment.

A month ago I hacked out a new tool – devstack-vagrant (available on github).

DevstackVagrant

Devstack vagrant provides a customized Vagrant environment that will build a 2 node devstack cluster under VirtualBox. The basic model is 2 devstack nodes (a controller and a compute) that bridge through a physical interface on your host. The bridged interface is set as the default route in the nodes so that 2nd level guests created on top of this 2 node devstack can route to the outside world.

The nodes start and build from official Ubuntu 12.04 cloud images, and are customized using the puppet provisioning support in vagrant. There are a few config variables you need to set in a config.yaml, including hostnames, bridge interface, and the password hash you want your stack user to have. Basically enough to bootstrap the environment and then run devstack from upstream git.

I added a bit of additional logic to the end of the normal devstack process that includes installing an Ubuntu 12.04 and Fedora 20 cloud image in your glance, injecting the ssh public key for the stack user into the keyserver, and opening up ping and ssh in all the security groups.

I still consider this an expert tool at this point, as in, if it breaks you get to keep all the pieces. However, this has been useful to me so far, and given the pull requests I got the other day, seemingly is useful for others as well. Patches definitely welcomed. And if it looks like more folks want to contribute I’ll happily move to stackforge.

One of the things I’d love to do is sort out a functioning libvirt backend for vagrant (there are 2, they are both a little wonky) because then the 2nd level guests could use nested KVM and not be ridiculously slow.

This tool has already proved very useful to me, so hopefully it will be useful to others as well.

The miracle of a billion cameras

Meteor Fall

It sounds like a remarkable story, almost unbelievable: Anders Helstrup went skydiving nearly two years ago in Hedmark, Norway and while he didn’t realize it at the time, when he reviewed the footage taken by two cameras fixed to his helmet during the dive, he saw a rock plummet past him. He took it to experts and they realized he had captured a meteorite falling during its “dark flight” — when it has been slowed by atmospheric braking, and has cooled and is no longer luminous.

via Norwegian Skydiver Almost Gets Hit by Falling Meteor — and Captures it on Film.

Part of what’s amazing about so many people recording things all the time on camera is we get to see things that we know must be, but no one has directly observed before. Like rocks falling from the sky.

I think XKCD sums it up best:

XKCD Settled

Neither Confirm Nor Deny – Radiolab

How a sunken nuclear submarine, a crazy billionaire, and a mechanical claw gave birth to a phrase that has hounded journalists and lawyers for 40 years and embodies the tension between the public’s desire for transparency and the government’s need to keep secrets.

via Neither Confirm Nor Deny – Radiolab.

Radiolab rarely disappoints, however this recent episode was pretty amazing. It tells the story of the origin of the “Neither Confirm Nor Deny” (aka Glomar) response that government agencies give to freedom of information requests.

Worth a listen.

Bundling is Worse

This isn’t just an Amazon problem. In the last few years, Google, Apple, Amazon, Facebook, and Twitter have all made huge attempts to move into major parts of each others’ businesses, usually at the detriment of their customers or users.3

Google, the geek world’s undeserved, unquestioned darling for well over a decade, has made all of its core products worse by forcefully shoving Google+ into them. They’re leveraging extreme success from some businesses (search, email, maps) to juice the numbers of one that’s faring poorly against its competitors (Google+). Sound familiar?

Apple’s Maps is still worse and has fewer features than Google Maps, which was previously integrated better into the iPhone and didn’t enable as much Google tracking creepiness. Not anymore. (Although I think the fault of this is shared between Apple and Google.) Many of Apple’s other applications and services have suffered as well as they’ve spread themselves too thinly and competed on more fronts.

via Worse – Marco.org.

I don’t usually read or link to Marco, because he’s got some very specific chips on his shoulder. However I think he hits the nail on the head here.

This vertical bundling of unrelated services that you can only really get if you buy into all of them is getting really frustrating. It’s actually one of the reasons I’ve always been anti-Apple, because the bundle was deep to their core, and I’m an a-la-carte kind of guy. But now everyone else is mocking Apple and trying to bundle all their services. Interop is being dropped left, right, and center.

You know who else loves bundles: your cable company.

This transition of tech services away from the pro consumer a-la-carte to monolithic bundles that mean you have to buy into a lifestyle to get any real benefits, is really frustrating.  You can no longer photo share from Picassa to anything other than G+, for instance, which was the last straw for me, and I’m getting all my photos out of Google now. It’s the reason I don’t buy much on VOD services, because I challenge you to find one that will let me play on a Linux Laptop, a Nexus 7 tablet, and a Roku (that’s my minimum set of devices I need to support).

It’s like the bad old internet with AOL, Compuserve, and Prodigy silos all over again.

So, I agree with Marco, this situation is worse. It’s also the sort of thing that AT&T, IBM, and Microsoft were taken to anti-trust for. But, like the cable companies, because there are multiple monopolies it will probably be seen as ok, even if it’s anti consumer. Like the cell phone market in the US.

But it’s still worse.

Various rambling thoughts from my personal corner of the internet