Mike Conigliaro

Disposable (Not Immutable) Infrastructure

People are addicted to the idea that there’s a simple answer for everything, and in our industry, that answer is currently immutability. The basic idea is that mutable state is a common source of defects in computer programs, and use of immutable objects can result in much simpler code with less defects (especially when it comes to concurrent programming).

Of course, the law of the instrument means people will want to use immutability to solve problems outside the world of software architecture too. Hence, we now have “immutable infrastructure”: the idea that you should never make changes to a running machine. If something needs to be changed, you should build a completely new machine, swap it in place of your old one, then destroy the old one.

This idea comes from the fact that it’s impossible to manage every single detail of a machine using current configuration management tools. This leaves open the possibility for unexpected changes on long-running machines, which can lead to problems caused by gradual configuration drift.

The Limits of Immutability

Unfortunately, with all our focus on the benefits of immutability, some of us seem to be missing the fact that immutability is only beneficial to a point. In software, pure immutability gives you nothing except a hot CPU. You eventually need to mutate the system in some way to do anything useful, even if it’s just to print something on a screen.

Similarly, stateless application servers probably aren’t very useful on their own, and since there’s no such thing as an immutable database, immutability is probably not something you’d want to enforce across your entire infrastructure. This means you end up with two different ways of managing your infrastructure. The immutable approach will work for truly stateless services, but you probably still need a way to manage long-running, stateful services anyway. So have you really solved a problem, or have you just made things more complicated?

Why Disposable is Better

Treating your infrastructure as disposable means accepting configuration drift as a possibility, then concentrating on making sure you can easily rebuild any machine from scratch when you want to. This is a more traditional way of using configuration management, and my preference for it basically boils down to having a consistent and more flexible way to manage my entire infrastructure.

For example, in the case of small, infrastructure-wide configuration changes, an immutable approach would require you to destroy and rebuild your entire infrastructure. On the other hand, a disposable approach means you can simply update your configuration management code and let the change slowly roll out on its own. Treating your infrastructure as disposable means you’re never forced to destroy and rebuild machines if you don’t want to, but you can still do it any time you need to.


Although I understand the problems immutable infrastructure is trying to solve, I frankly have yet to experience them in any significant way. My sense is that when you’re working with good people who know how to use their tools, problems caused by configuration drift should be rare. And even if these problems do occur, proper use of configuration management means there’s nothing stopping you from destroying and rebuilding machines any time you want. And that’s my main complaint about immutable infrastructure. Its ideas seem to be purely subtractive; removing tools from my toolbox as if I can’t be trusted to use them responsibly.

Incidentally, when I see people going as far as suggesting things like disabling SSH and using random/obscure hostnames to make logging into individual machines more difficult, I have to wonder if they’ve ever had to troubleshoot anything in a real production environment (Hint: A centralized logging architecture is great, but by no means a substitute for tools like strace, lsof, and tcpdump).

The Problem with Other People's Infrastructure Code

We have a problem with instant gratification in our industry. I first wrote about this back in 2008 when I complained about wizards, but these days, I’m more concerned about shared infrastructure code (scripts, Chef cookbooks, CloudFormation templates, etc.). With the proliferation of configuration management systems and similar tools, it’s now quite possible to copy and paste your way to a functional production environment in minutes. Unfortunately, I believe all this instant gratification comes with hidden but significant long-term costs, and it needs to be recognized for what it (usually) is: technical debt.

Lack of Control Means Lack of Consistency

A lot of people like to think that shared infrastructure code is like software libraries; that it’s composable and can be swapped in and out with little forethought. The reality is often far from that, forcing you to accept other people’s infrastructure decisions in a way that can be very difficult to change without a series of painful migrations.

Go down this road enough times and you’ll realize that it’s basically impossible to enforce any kind of consistency across your infrastructure when your implementations come from different authors who are not beholden to your organization, and who have wildly different opinions about the “correct” way to do things. This lack of consistency can very quickly turn on-boarding and general management of your environment into a nightmare.

Increased Complexity

For some, the solution to the inconsistency problem is an attempt to create something configurable enough to work for everyone. Unfortunately, this leads to exponential increases in code complexity as the author inevitably tries to support every possible configuration on every operating system known to man (I’m fully convinced this is how Chef got its undeserved reputation for being complicated). This increased complexity often leads to more bugs and more time spent troubleshooting, which will begin to negate the time you thought you’d saved in the first place.

On Reinventing Wheels

Being able to push a button and get a cluster of machines in a few minutes is only the beginning of a good implementation. What will it take to maintain that cluster over the long-term? How do upgrades work? How do you recover from a node failure? How do you avoid getting woken up in the middle of the night? These are the kinds of things every good operations person is thinking about before introducing a new technology to their environment, and having infrastructure code that complements the long-term plans is an integral part of the solution. The alternative is akin to closing your eyes and hoping that some random person on the Internet’s ideas perfectly align with yours.

In this case, “reinventing the wheel” is far from a waste of time, because it’s probably the only way to get a wheel with your exact specifications. More importantly, since reinventing wheels is how we learn to build and maintain wheels, any time spent here will pay huge dividends later in the form of knowing what to do when something goes wrong.


While shared infrastructure code makes for useful examples and as a quick way to test drive things, it’s probably not the best building material for your production environment. Far different from composable software libraries, careless use of other people’s infrastructure code is more akin to forking a web application that looks similar to what you want rather than starting from scratch and building the simplest thing that will work for you. The former strategy will most likely result in a mess, while the latter will result in a codebase that is far smaller, simpler, more consistent, and easier to maintain over the long-term.

My Essential Sublime Text Plugins

Sublime Text has been my go to editor since John De Goes introduced me to it around 2010, during our time at SocialMedia.com. I had already been looking for a new cross-platform editor to replace my beloved jEdit, which even then was feeling a bit old and neglected. It took a year or so before I was able to fully make the switch (Sublime Text was in pretty heavy development at the time), but these days, you can find a plugin to help with just about anything.

Package Control

Package Control is the package manager for Sublime Text, and the main Package Control website is the place to search for new plugins.

Once it’s installed, type ctrl+shift+p (Win, Linux) or cmd+shift+p (OS X) followed by package (all Package Control commands begin with “Package Control:”).

Also see syncing for instructions on how to sync your installed packages across different machines.


SublimeLinter is a lint framework for Sublime Text. Once it’s installed, you’ll be able to use several lint plugins that make use of it. These plugins have saved me from countless errors.


GitGutter shows you what lines were changed, added, or removed since your last commit. It’s like having a constantly updated git diff built in to your editor.


There’s nothing worse than updating a README.md and not realizing that your formatting is all screwed up until you look at it on GitHub. The MarkdownEditing plugin gives you some nice Markdown syntax highlighting, so you can preview your changes before committing them. Like the SublimeLinter plugins above, this one has saved me from countless errors.


The context menus in Sublime Text’s sidebar are pretty limited by default. The SideBarEnhancements plugin adds lots of useful file operations.


The Chef plugin hasn’t been updated in a long time, but it’s still useful in Sublime Text 3. It provides auto-completions for most of Chef’s built-in resources, which makes writing cookbooks faster and easier.

Troubleshooting Headless Tests on a Remote CI Server

I just ran into a problem that was causing the Jasmine tests on our Jenkins CI box to hang forever, and I figured I should document this handy little troubleshooting tip in case someone else might find it helpful.

If you hop onto your CI box while your headless browser tests are running, you should see an Xvfb process that looks something like this:

# ps -efww | grep Xvfb
jenkins  18833     1  1 21:18 ?        00:00:00 /usr/bin/Xvfb :99 -screen 0 1280x1024x24 -ac

Since Xvfb is using display :99, you’ll want to run x11vnc accordingly:

$ x11vnc -display :99

Now you should have a VNC server listening on port 5900. Just fire up your VNC viewer and connect as usual:

$ vncviewer host.example.com:5900

But what if you’re accessing your CI server over an insecure network? You can use SSH local port forwarding to create a secure tunnel:

$ ssh -L 5900:localhost:5900 host.example.com x11vnc -display :99

Now you can connect to the VNC server over the secure tunnel:

$ vncviewer localhost:5900

From WordPress to nanoc

I finally decided to start blogging like a hacker, which is ironic, considering that I’ve actually come full-circle now. Back in the late-nineties (before I knew about databases, and before the term “blog” even existed), I spent a lot of time working on a Perl-based blogging engine that actually worked pretty similarly (though much less sophisticatedly) to how a lot of today’s static site generators work. Instead of working with sane formats like YAML or Markdown (which didn’t exist back then), I ended up managing everything in my own pseudo-XML format. As ugly and hackish as this system was, the resulting output was pretty nice, and the project was definitely a good learning experience for me as a teenager. But now that static site generators are all the rage, there are a lot of much better options out there, which gives us “hackers” a good opportunity to migrate away from WordPress without a ton of effort.

After reading this thread on Hacker News about yet another new static site generator, I decided to give nanoc a spin (mostly because it seemed to be the most popular option by people commenting on that thread). I skimmed through the nanoc Getting Started guide and (for the first time in several years) set out building a new layout for my new site. Since I didn’t want to revisit the pain of the old days of web design, I made sure to get Compass integration working right away.

Once the layout was done, I spent about a week slowly reimplementing features and moving data over from my old WordPress site. The static pages were basically just a copy & paste (with some manual converting to Markdown), but there was no way I was going to repeat that process for 100+ blog posts. There are a few example WordPress-to-nanoc scripts floating around, but they all left a lot to be desired, so I ultimately ended up writing my own. The result of that effort can be found in wp2nanoc.rb. Besides the addition of some nice command line option parsing, my script also does some basic conversion from HTML to Markdown and from WP-Syntax to SyntaxHighlighter.

The last thing to bring over were my comments, and Disqus handled most of that for me. I basically just installed the Disqus WordPress plugin and ran through the automatic import process. Then to make the comments show up on my new site, I set disqus_shortname and disqus_url to the appropriate values in my embed code. Note: If you find the developer documentation as confusing as I did, just know that these are the only values that need to be set. I originally tried using disqus_identifier, but that didn’t work, because the plugin uses unconfigurable, WordPress specific values for this option which obviously won’t be available in nanoc.

So what did I gain from my migration to nanoc?

  • I can now keep my entire site (data and all) in git
  • Static files mean blazing speed
  • No database means I can host my site in Amazon S3 for pennies
  • No more worrying about PHP/WordPress security issues
  • I got to experiment with new (to me) technologies like Haml, Sass/SCSS, Compass and Blueprint