Saturday was a rainy day, so I decided to deal with switching out the root disk on my home server with an SSD that I purchased a couple weeks ago. It’s part of my quest to get all the root disks of my machines off spinning media. My home server was a build from parts machine, that’s long enough in the tooth that it won’t boot from USB. So I found a stack of CDRs upstairs, of equally dubious age, burned an Ubuntu 12.10 server iso, and started the install.
Things were chugging along quite well until the installer was supposed to install additional packages. Then it bombed out (I blame the ancient CDRs). I was able to get it to at least install grub, and get the thing to boot back onto the network.
What I found myself with was a super minimal install. It didn’t yet have a normal sources.list, it didn’t have openssh-server, it didn’t have ssh client even, it didn’t have any of the normal even minimal server install tools. I had about 30 minutes of manual to typing to get the base apt repo in, and get me so I could ssh in from upstairs to drive the rest of the process.
Boostrapping a Puppet Master
This is the machine that’s my puppet master. I had a copy of the oldroot over in one of my software raid arrays, so the moment I got that mounted, I dumped over the /etc/puppet this machine should have, and tried to just puppet my way up the rest of the way. I’d been on a month long kick to puppetize my home infrastructure, so this was a promising direction.
It turns out puppet up from nothing is a little harder when you are the puppetmaster, and dnsserver for the network as well. 🙂 So it was about another 30 minutes of manually installing what was needed to get my puppetmaster started. Once that was up, I managed to get the first puppet agent run in, and it was epic. 45 minutes chugging away pulling down all the policies I needed, applying packages and configs, all the kind of magic that prevented me from spending my whole day trying to figure out how I had this server setup before.
It also showed me where my policy had holes. I’ve got xfs filesystems now, so xfsprogs need to be in the base case. My libvirt setup didn’t actually install kvm, but in the super minimal install, that wasn’t there. I hadn’t gotten around to managing my openvpn server yet, that’s in there now.
If I was to do it again…
One thing I really need is both a puppet and puppetmaster bootstrapping script. Using puppet to manage your puppetmaster is cool and all, but there is a bit of snake eating it’s own tail to get you started that required a little more manual command slinging than I liked.
But, had I not had so much of my server policy encoded in puppet, I’d still be typing commands now to get that box up and running. So I’m sold on the whole process, even for a smallish IT environment, like a few home servers and remote guests.
3 thoughts on “How puppet rescued my botched server install”
I’ve recently started using puppet to manage my home boxes as well, but I haven’t gone the full puppetmaster route quite yet: I’m using https://github.com/brightbox/puppet-git-receiver. It has the drawbacks you could imagine — namely those associated with not having a puppetmaster — but for my purposes it gives me many of the benefits of puppet without the overhead of a “real” setup.
It’s not all that much effort to setup a puppet master. You can even have puppet do it for you. 🙂
Honestly the cert signing is the only bitch of it, but if you want to deploy stuff with passwords, you kind of need to do that so you can hide the secrets in hiera.
…though now that I think about it, you may be able to use a subset of your full config to bootstrap the master via the receiver. I do a variant of that, actually; first run is always a minimal manifest/node config for me that configures and installs the puppetlabs repo and puppet; second is the “real” config.