We moved in to our current data center (365 Main) more than two and a half years ago in preparation for taking on hosting and server management at a larger, more professional scale. The basic shape of our current infrastructure — the high-level design of our networking, monitoring, provisioning, update, backup and other components — was largely formed at that time. Most components have seen a great deal of refinement in that time, but with hindsight there are a number of things we’d do differently.
One of the things we’d do differently is that we’d get a data center contract longer than three years. Sure enough, 365 Main is now more or less full and new contracts have per-rack power limits well under half of what we use. So while we may (or may not) maintain some presence there, it’s essentially not an option to keep our full infrastructure at 365 Main beyond the end of our current agreement (the end of March).
For the last several months Michael has put an extraordinary amount of effort into identifying and evaluating prospective data centers, in the San Francisco bay area and beyond. When it comes down to it, none of the bay area data centers has the power density that we want and all of them involve significant compromises. We’ve even considered building out our own (or rather, improving a facility that’s 80% of the way to being a data center), but that approach didn’t really fit our model — facilities management is best left to people who specialize in it (and new data centers always have more than their share of problems).
Since Michael and Linda will be moving to the Seattle area next spring, that area (which also happens to have relatively cheap power) merited special consideration. Network-wise it’s essentially just as good as the bay area, it isn’t prone to earthquakes (or hurricanes, etc.) and the data center market has some better (and more power-dense) options. We haven’t finalized things, but it looks like we’ll be moving most of our infrastructure to a data center in Seattle (but probably keeping some space in the bay area and starting to build a redundant-site architecture).
Regardless of our exact choice, the silver lining in having to move is that we’ll have an opportunity rebuild the entire infrastructure from scratch. Over the next two months we’ll build the core components in a staging environment here, and then we’ll install them plus a bank of new servers in the new data center. For most of our customers, we’ll build a complete parallel version of their site (which can be thoroughly tested in advance), keep the data nearly in sync for a while, and then do a final sync-and-switch with just a few minutes of downtime. This will also allow us to update all of our customers to our latest configuration standards.
Much of this is the boring work of making all of the loose ends tidy (such making remote power control more uniform and fixing some path inconsistencies), but there’s also some fun stuff in building this new generation of our infrastructure. We’ll put in SAN storage (used sparingly at first, both as a better answer than dedicated file servers for sites needing more than 0.5TB but less than 5TB and to support virtualization with fast migrations), implement BGP (perhaps using two Internap PNAPs as uplinks) and, as always, implement ever more automation. (We’re still exploring our SAN options, so if you have thoughts on iSCSI/AoE options with SATA disk in the 10-50TB range, please leave a comment.)
Of course, we’re more than busy enough already, but when aren’t we?
