Dirk & I will be in Seattle this week to do some prep work on our new datacenter. Our schedule is a bit strange, landing at 9am, taking off at 9pm, and a datacenter meeting from 10am-1pm.

We’re free from 1pm - 5pm if anybody wants to squeeze in coffee, or perhaps point out some great photo ops, let me know! We’re dining with some friends at Ocean Aire around 5:30.

My Canon 400D gets in tomorrow, and I expect to come back thursday with a flashcard filled with datacenter pictures to post.

Well, it’s official! We signed a 7-year contract for 300 square ft of datacenter space at Fortress in Seattle, WA. At the very least, I can say that I’m very excited about this new facility. I have a really good feeling about the owner, the facility itself, and the location. Our deal is pretty sweet as we’re their second customer, so the economics of the place are far better than anything in the bay area.

Over the next few months I’m hoping to detail all of the steps involved with rolling out our new datacenter. The first two steps, which were reconnaissance & signing a contract, are finally complete. The next two steps are design & purchasing. I’ll be flying up the middle of next week with Dirk to draw out the space, decide on our cabinet layout, and put together a solid diagram from which to create a purchasing list.

Somehow it slipped my mind, but this past Tuesday noted the one-year anniversary of a customer of ours, League Of Professional Systems Administrators (LOPSA).

LOPSA is a great customer of ours, and was our first foray into charitable contributions. It all started September, 2005. The SAGE board had been commissioned to split away from Usenix a year before, and things started falling apart in this process. Aug 2005, I was approached by (I believe, forgive me if I’m wrong) Stephen Potter about providing infrastructure to SAGE as they split off from Usenix, to which we tentatively agreed.

Linda, Daniel & I sat down, and came up with an appropriate offer for their needs. We would provide them with two fully managed servers, and treat them like any other customer of ours, except that we would be donating the management costs. Just a couple of weeks after this had been deployed, the SAGE split-off dissolved, and we promised to let them continue using the servers until resolution, or until a new body had been formed.

The servers we provided LOPSA with were BitPusher’s standard servers, the Proliant DL145 G2 from Hewlett Packard

On November 14, 2005 LOPSA officially came to life. Since then, LOPSA has grown to a membership of over 600 members. It threw it’s first training event in November, 2006, a week ahead of it’s one-year anniversary. It supports a very active community, including several very active mailing lists, a community site and provides it’s customers with many exclusive discounts.

It’s been a great year, so far. The board & volunteers at LOPSA have been a real pleasure to work with. They’ve even referred a couple of key customers to BitPusher, for which we are ever thankful! Happy Birthday LOPSA!

I was just surfing around Flickr and ran accross some pictures of Daniel’s sub-presentation at Carson Workshop’s summit, The Future Of Web Apps. They’re pretty good shots, so I thought I’d share. The first two are from Scott Beale / LaughingSquid.com, The last is from the flickr user Maqroll.

While poking around our website, I ran into some old photos from July, 2005 we had taken at our datacenter by Jake Appelbaum. I thought I’d post a few of them.

Here I am trying to look mean on top of 365 Main.

Left to right: Linda, Michael, Daniel trying to look tough.

(more…)

Last night, at 8:46 PM I sent an instant message to our awesome contract web designer, Justine from IndustriousBird.com asking about making WordPress look like the rest of our website. She merely asked for a login to wordpress, and said she’d look into it.

This morning, I awoke to a happy message from Justine merely stating “go to http://blog.bitpusher.com/ and let me know what you think”. Within 3 hours she had transformed our generic wordpress template into an exact replica of the rest of BitPusher’s website. Such a great way to start my morning!

Most days, everything runs smoothly. The monitoring system stays green, you push out new OS patches and watch the automated patch management apply them without error. The power stays on, the bits keep flowing, support tickets are closed quickly, and all is good.

On other days, however, forces seem to conspire against you. This week we’ve got one of those days brewing. Our datacenter discovered that they had oversold, and underprovisioned power in the room where our networking equipment is. They’ve decided that they need to bring down the 6 power circuits that keep all of our networking equipment running, because they need to replace two Power Distribution Units to prevent a possible failure.

All of the planning we put into preventing these issues almost feels superfluous. We have redundant routers, switches, and firewalls at that layer. We have redundant power in these devices. Even the power supplies are on switches that go to PDUs at opposite ends of the colo room. Foiled by the fact that our provider decided they needed to take both PDUs down at the same time.

So, we get to implement contingency plans. In a careful rush, we’ve lowered the TTLs of all of our customers’ DNS names. We provisioned a server in another datacenter to be a mail relay, to make sure no email gets losts. We’ve also setup a webserver on this site, and are redirecting customer site to this server via DNS, to let our customers’ customers know that they haven’t dropped off the face of the earth.

Then we send out the notification to our customers, and plan for a long night. Tonight is going to be a very long night, here’s the notification.

To: Customer
Subject:BitPusher Services maintenance notification:
————————————————————————

ESTIMATED START TIME: 11/03/2006 21:00 PST
ESTIMATED END TIME: 11/04/2006 01:00:00 PST

SERVICES AFFECTED: All
TYPE OF WORK: Power work
PURPOSE OF WORK: Emergency power repair.
IMPACT OF WORK: All network services will be unavailable during the
repair work.

Our colo provider, 365 Main, alerted us that they need to perform emergency work on their power system tomorrow night. All of our equipment is on redundant power feeds to two separate Power
Distribution Units, but unfortunately 365 will be bringing down both PDUs that we are connected to.

We are taking measures to minimize the effect of this outage. We have setup a webserver and mail relay on an outside network, and will be redirecting website hits and e-mail traffic to this location for the duration of the outage. We ask that if you have a specific maintenance page that you would like us to post, that you would e-mail it to support@bitpusher.com asap.