Most days, everything runs smoothly. The monitoring system stays green, you push out new OS patches and watch the automated patch management apply them without error. The power stays on, the bits keep flowing, support tickets are closed quickly, and all is good.

On other days, however, forces seem to conspire against you. This week we’ve got one of those days brewing. Our datacenter discovered that they had oversold, and underprovisioned power in the room where our networking equipment is. They’ve decided that they need to bring down the 6 power circuits that keep all of our networking equipment running, because they need to replace two Power Distribution Units to prevent a possible failure.

All of the planning we put into preventing these issues almost feels superfluous. We have redundant routers, switches, and firewalls at that layer. We have redundant power in these devices. Even the power supplies are on switches that go to PDUs at opposite ends of the colo room. Foiled by the fact that our provider decided they needed to take both PDUs down at the same time.

So, we get to implement contingency plans. In a careful rush, we’ve lowered the TTLs of all of our customers’ DNS names. We provisioned a server in another datacenter to be a mail relay, to make sure no email gets losts. We’ve also setup a webserver on this site, and are redirecting customer site to this server via DNS, to let our customers’ customers know that they haven’t dropped off the face of the earth.

Then we send out the notification to our customers, and plan for a long night. Tonight is going to be a very long night, here’s the notification.

To: Customer
Subject:BitPusher Services maintenance notification:
————————————————————————

ESTIMATED START TIME: 11/03/2006 21:00 PST
ESTIMATED END TIME: 11/04/2006 01:00:00 PST

SERVICES AFFECTED: All
TYPE OF WORK: Power work
PURPOSE OF WORK: Emergency power repair.
IMPACT OF WORK: All network services will be unavailable during the
repair work.

Our colo provider, 365 Main, alerted us that they need to perform emergency work on their power system tomorrow night. All of our equipment is on redundant power feeds to two separate Power
Distribution Units, but unfortunately 365 will be bringing down both PDUs that we are connected to.

We are taking measures to minimize the effect of this outage. We have setup a webserver and mail relay on an outside network, and will be redirecting website hits and e-mail traffic to this location for the duration of the outage. We ask that if you have a specific maintenance page that you would like us to post, that you would e-mail it to support@bitpusher.com asap.

Perhaps the weight is one ton. Maybe it’s five. Either way, it’s off my shoulders at last! Nearing the end of a project (even if it means another, larger project is starting very soon) is one of the best feelings in the world.

For the past four months, a large percentage of my efforts have been towards procuring new datacenter space for BitPusher. This is the 14th or 15th time in my life that I’ve gone through the datacenter evaluation process. It’s only the 2nd time that I’ve done this for my own company though, so it was probably the most important.

Now that the selection phase is over, I’m going over my journal, notes, timelogs, and loose ends that emerged this journey. So much work went into this process that I am teaching a class about this for LOPSA’s, SysAdminDays” in Phoenix, on November 7th. The class is entitled “Data Centers, Data Closets and Server Hosting”.

I thought I’d share some fun statistics related to this effort.

* E-mails sent: 478
* E-mails received: 536
* Quotes received: 14
* States visited: 2
* States datacenters are located in: 8 (Pennsylvania, Nevada, Iowa, Idaho, Washington, Arizona, Chicago, and California)
* Time on telephone: 26 hours, 40 minutes
* Miles flown: 2,500
* Datacenters visited: 11
* Datacenters spoken to: 19
* Overnight stays: 5
* Quote revisions: mathmatics fails to provide me with an accurate estimate

Once the contract is signed, I’ll write a quick follow-up espousing the many virtues of our new facility.

We moved in to our current data center (365 Main) more than two and a half years ago in preparation for taking on hosting and server management at a larger, more professional scale. The basic shape of our current infrastructure — the high-level design of our networking, monitoring, provisioning, update, backup and other components — was largely formed at that time. Most components have seen a great deal of refinement in that time, but with hindsight there are a number of things we’d do differently.

One of the things we’d do differently is that we’d get a data center contract longer than three years. Sure enough, 365 Main is now more or less full and new contracts have per-rack power limits well under half of what we use. So while we may (or may not) maintain some presence there, it’s essentially not an option to keep our full infrastructure at 365 Main beyond the end of our current agreement (the end of March).

For the last several months Michael has put an extraordinary amount of effort into identifying and evaluating prospective data centers, in the San Francisco bay area and beyond. When it comes down to it, none of the bay area data centers has the power density that we want and all of them involve significant compromises. We’ve even considered building out our own (or rather, improving a facility that’s 80% of the way to being a data center), but that approach didn’t really fit our model — facilities management is best left to people who specialize in it (and new data centers always have more than their share of problems).

Since Michael and Linda will be moving to the Seattle area next spring, that area (which also happens to have relatively cheap power) merited special consideration. Network-wise it’s essentially just as good as the bay area, it isn’t prone to earthquakes (or hurricanes, etc.) and the data center market has some better (and more power-dense) options. We haven’t finalized things, but it looks like we’ll be moving most of our infrastructure to a data center in Seattle (but probably keeping some space in the bay area and starting to build a redundant-site architecture).

Regardless of our exact choice, the silver lining in having to move is that we’ll have an opportunity rebuild the entire infrastructure from scratch. Over the next two months we’ll build the core components in a staging environment here, and then we’ll install them plus a bank of new servers in the new data center. For most of our customers, we’ll build a complete parallel version of their site (which can be thoroughly tested in advance), keep the data nearly in sync for a while, and then do a final sync-and-switch with just a few minutes of downtime. This will also allow us to update all of our customers to our latest configuration standards.

Much of this is the boring work of making all of the loose ends tidy (such making remote power control more uniform and fixing some path inconsistencies), but there’s also some fun stuff in building this new generation of our infrastructure. We’ll put in SAN storage (used sparingly at first, both as a better answer than dedicated file servers for sites needing more than 0.5TB but less than 5TB and to support virtualization with fast migrations), implement BGP (perhaps using two Internap PNAPs as uplinks) and, as always, implement ever more automation. (We’re still exploring our SAN options, so if you have thoughts on iSCSI/AoE options with SATA disk in the 10-50TB range, please leave a comment.)

Of course, we’re more than busy enough already, but when aren’t we?