Daniel and I are in Chicago for HostingCon! To read  about our first day’s adventures click here. I’ll be blogging about HostingCon a few times per day at my NetworkWorld blog, DatacenterJunkie. Amongst some of the  highlights will be interviews with hosting industry leaders, almost-live coverage of the interesting tracks, and photos of the conference.

Daniel and Michael will be in Chicago July 27th through August 1st for HostingCon. We’ll be wandering around in BitPusher shirts, so make sure to grab us and say hi if you’re at the conference. Also, if you’re in town, leave a comment and we’d love to take you out for dinner or drinks this week.

To all BitPusher customers,

Our data center has informed us that they need to do maintenance on the power system in the next few days in order to prevent an unscheduled outage in the future. They have schedule an outage which will require us to take down all services based in our main (Seattle) data center.

The time during which BitPusher customer services will be down is (in PDT — see below for other time zones):
Thursday, September 14 18:50 to 19:30 (40 minutes)

All services based on our Seattle data center will be shut down at 18:50. This will be a clean shutdown. As soon as the power work is complete we will restore services. All services will be restored by 19:30.

We apologize for any inconvenience this will cause. Please let us know if you have any questions of if there is anything we can do to mitigate the impact for you and your customers.

Here is the start time of the outage in other time zones:
Friday, September 14 01:50 UTC
Thursday, September 13 18:50 PDT (US West Coast)
Thursday, September 13 21:50 EDT (US East Coast)
Friday, September 14 02:50 BST (United Kingdom)
Friday, September 14 03:50 CEST (France)
Friday, September 14 11:50 Aus. EST (Eastern Australia)

Michael T. Halligan
————————
Chief Technology Officer
888-978-7437
BitPusher, LLC
http://www.bitpusher.com/

As expensive as it was, it was absolutely the right system to move BitPusher’s critical infrastructure off of California’s over-burdened power grid.

Apparently a transformer exploded under the 560 Mission PG&E Substation in San Francisco, knocking out part of the grid for about 24,000 people. Unfortunately, this happened in the SoMA neighborhood, otherwise known as datacenter alley where 365main, JMA Wired, ServePath, 650 Townsend, and Level3 (amongst others) have datacenters. Apparently the datacenter we moved out of, 365main, had some problems dealing with this power outage.

• Power Outages In San Francisco
• 365Main datacenter power outage
  
• At least 20,000 without power in San Francisco

Technorati Tags: 365main, pg&e, pge, datacenters, colo, power outage

To the extent that things at BitPusher ever settle down, things have settled down after the move to Seattle.

We’re already seeing benefits from move. One big networking snafu notwithstanding, the combination of better and beefier shared infrastructure with the opportunity to rebuild our customer servers has noticeably reduced the frequency of operational incidents. In practical terms, this has meant fewer pages for our operations team and fewer process restarts — things that most people don’t notice, but they take up our time and can cause brief downtimes or cause end user sessions to be lost.

But perhaps the biggest difference is in the little ways we’ve made the infrastructure simpler and tidier. It may sound silly to dictate that the server in slot 13 in cabinet 2 will be known as sea-c02-s13 and be must be plugged in to port 13 on the both the remote-control power system and each of the three network switches to which it is connected (two redundant data connections plus one for the lights-out management), but in the end it saves time and avoids confusion, as well as eliminating the need to maintain much of a cable map.

Rebuilding customer servers, as we did during the move, has meant that everything (well, almost everything) is now set up according to our latest best practices. While we always kept the servers up to date in terms of patches and such that affect security and stability, many of the details of how we configure networking, choose file paths, etc. have changed in the three years since we moved in to our previous data center, and some of those are harder to update. Rebuilding gave us a chance to update them. (We’ve also learned a lot about what to do and not do when rebuilding customer environments, so it will be easier to do limited rebuilds to keep things up to the latest spec in the future.)

Improvements in the physical data center space make a big difference, too. In addition to better power and cooling and having plenty of room to expand, it helps a great deal to have better cable management, a simplified and easier-to-manage scheme for labeling servers, and more storage space for easier management of spare parts.

When trying to keep a complicated environment running smoothly, the little things make a big difference.

As of 9:00 PM on May 17th, BitPusher has finally moved 100% of it’s infrastructure out of 365 Main in San Francisco. It’s funny, we still have access to 1/2 of the colocation rooms in 365, since we still manage a handful of customers there, but we no longer maintain any of our own equipment there.

It’s good to be completed with this project. Our new environment is a well-oiled machine. Essentially, we started with a green-field environment, and put together a long list of gripes about our old infrastructure in San Francisco. Everything from cabling standards to setting up BGP, and building better systems management tools was addressed.

More to come in the next week, but after such a lull in blogging activity, I thought I’d make this exclamation. We’re open, and doing business in Seattle!

Jr. Admin CONTRACT (Seattle, WA) January 9th through January 12th

BitPusher, LLC is seeking a contractor to assist in the build-out of our new datacenter in Tukwila.

This is a short-term contract from January 9th, 2007 through January 12th, 2007 (around 25 hours), which may lead into an ongoing part-time NOC technician position.

Responsibilities will include:

* Installing rack-mount equipment including servers, switches, routers
* Installing ethernet cables
* Documenting cabling infrastructure
* Upgrading BIOS & Firmware on servers
Qualifications:

Requirements:

* 1-2 years Linux Administration Experience
* Able to work up to 4x 10 hour shiftts from 8am - 6pm
* Ability to lift 60lbs overhead; may require awkward or straining position to be assumed for short periods of time in lifting or han
dling objects
* Experience running ethernet cable
* Familiarty with Linux
* Strong work ethic
* Reliable transportation
* Experience working in datacenters

Compensation: DOE, 1099

BitPusher, LLC is a rapidly growing IT managed services startup. Please contact us by e-mail only. Please send a resume and cover letter to hr@bitpusher.com.

Our hardware order has finally been placed! It’s been a good solid month of price competition, quote modification, and lease arrangements. As of about an hour ago the purchase orders have been issued from our leasing companies to our hardware vendors. The bulk of our hardware order is being placed through CDW thanks to their cut-rate pricing. The rest of our hardware is a couple of Cisco 6506s from Recurrent, a Cisco reseller based out of Santa Clara.

Dirk & I are heading out to Seattle from January 8th -> January 15th, so if anybody would like to get together during that period, let us know. We’ll be working furiously racking hardware, running cables, and building our new network, but we’re trying to set our evenings aside for dinners with friends, colleagues & customers, plus we’re going to spend Saturday & Sunday exploring if we make enough progress.

We began our datacenter deployment “officially” three weeks ago by signing with Fortress in Seattle. A week later, Dirk & I flew to Seattle to survey our new space and meetup with a sales lead. For posterity’s sake, I photoblogged our journey.

Our day began with taking off from Oakland Airport at 7:20 AM, arriving into SEATAC at around 9:30 AM. We picked up our Cadillac DTS at about 10:15 (we figured if we’d be driving around a wet, cold, rainy, unfamiliar city all day we might as well be in something huge, unwieldy, and comfortable) and headed to the datacenter.

(more…)

Well, it’s official! We signed a 7-year contract for 300 square ft of datacenter space at Fortress in Seattle, WA. At the very least, I can say that I’m very excited about this new facility. I have a really good feeling about the owner, the facility itself, and the location. Our deal is pretty sweet as we’re their second customer, so the economics of the place are far better than anything in the bay area.

Over the next few months I’m hoping to detail all of the steps involved with rolling out our new datacenter. The first two steps, which were reconnaissance & signing a contract, are finally complete. The next two steps are design & purchasing. I’ll be flying up the middle of next week with Dirk to draw out the space, decide on our cabinet layout, and put together a solid diagram from which to create a purchasing list.