Network Outage [ThePlanet/Houston1-DC]

[Cranky]

At this rate give it by tomorrow morning, we were at our last surge of possible traffic for today before tomorrow afternoon/evening. Not big, to me. Besides I wanted to get a few sites up again today, oh well.

[GvilleRick]

BTW, Serra, saw one of your posts in The Planet's forum. It was a bit disconcerting to see the Newbie label there.

[gwyneth]

Originally Posted by GatorEricS

At this time, the outage of Houston1-DC is affecting approximately 40 dedicated customers (of which we are aware), most of which are on our unmanaged dedicated plans*. This is not currently affecting any of our shared or reseller servers. Our monitoring and critical internal systems remain online at this time.

Hey, DediKiddy(tm) owners are cool.

Originally Posted by pbradish

Not good. I have clients who are losing thousands because of this and are already requesting new server hosts.

With that much at stake, it's a good thing you took daily backups and had a contingency plan ready to go.

[GvilleRick]

I agree. I check out cPanel's forums pretty regularly and post occassionally but there are so many ridiculous questions that I lose interest.

[GatorEricS]

Hi Gwyneth, I apologize, I wasn't trying to make DediKiddy(tm) owners seem less than cool. I was merely specifying dedicated plan names to make determining whether or not one was affected, easier. In fact, we have a significant number of unmanaged customers, a lot of the newer ones have been referred (by me, and some of my co-workers) to your DediKiddy(tm) Threads post. I can't even begin to explain how helpful that has been.

[Cranky]

I'm well aware, I might not sleep tonight.

[pbradish]

Originally Posted by gwyneth

With that much at stake, it's a good thing you took daily backups and had a contingency plan ready to go.

That much is at stake, only because the particular sporting even in our market is on CBS (for the first time ever) as we speak. Very, very bad timing and an unfortunate learning experience.

[Serra]

Originally Posted by pbradish

That much is at stake, only because the particular sporting even in our market is on CBS (for the first time ever) as we speak. Very, very bad timing and an unfortunate learning experience.

Right, if you've never experienced anything like thing, then yea, its a huge wakeup call. Servers, provided by Hostgator/ThePlanet will never have 100% uptime. We don't even get the luxury of having an SLA. (Well, if you are with thePlanet directly, you get a network SLA). A server with an SLA is MUCH more expensive. If that is what you really need, then this isn't the right place for you to be hosting.

However, being down while everyone is actually watching TV, isn't a bad thing. Being down on Sunday would be far worse!

If you need to, you can sign up for a shared account, change your name servers to the shared account and tell everyone what is wrong. The total cost of that is like $10.00. (Advice is dependent on your registrar. If you are with someone who has a quick TTL, like goDaddy, that works, if you are with a crap registrar, don't bother)

[pbradish]

Originally Posted by Serra

Right, if you've never experienced anything like thing, then yea, its a huge wakeup call. Servers, provided by Hostgator/ThePlanet will never have 100% uptime. We don't even get the luxury of having an SLA. (Well, if you are with thePlanet directly, you get a network SLA). A server with an SLA is MUCH more expensive. If that is what you really need, then this isn't the right place for you to be hosting.

However, being down while everyone is actually watching TV, isn't a bad thing. Being down on Sunday would be far worse!

If you need to, you can sign up for a shared account, change your name servers to the shared account and tell everyone what is wrong. The total cost of that is like $10.00. (Advice is dependent on your registrar. If you are with someone who has a quick TTL, like goDaddy, that works, if you are with a crap registrar, don't bother)

Thank you for the helpful tips Serra. We may have to start looking into other options (or at least start to research them now).

[gwyneth]

Originally Posted by GatorEricS

Hi Gwyneth, I apologize, I wasn't trying to make DediKiddy(tm) owners seem less than cool. I was merely specifying dedicated plan names to make determining whether or not one was affected, easier. In fact, we have a significant number of unmanaged customers, a lot of the newer ones have been referred (by me, and some of my co-workers) to your DediKiddy(tm) Threads post. I can't even begin to explain how helpful that has been.

Gosh, GatorEricS, I don't know why you're apologizing--I didn't think you were saying anything at all about us. What I meant was, DediKiddy(tm) owners are cool about s*** happening and may be least likely to harass you poor guys about this. So I'll apologize to you for phrasing things in such a way...the last thing you need at a time like this.

[DarkSorrow]

Originally Posted by Serra

Yea, its not that hard to setup a backup site, I mean I posted a way to do it for about $100.00 a year for shared users, cheaper maybe.

I'll tell you, if I stood to lose thousands in a few hours of downtime, I'd be triple redundant....


Oh, and just to be clear, my company website is running on the backup right now, so it's not down. My server is getting cold, and my website is up....

Never did see that post. And I thought I read em all

[Serra]

Originally Posted by pbradish

Thank you for the helpful tips Serra. We may have to start looking into other options (or at least start to research them now).

For fail-over, users can setup a dyndns account and fail-over to a shared server. (This is on a per account basis, so not for the whole server, but for critical accounts.)


It also helps for owner/resellers to know their own/customer's failure tolerance. How much will an outage of 12 or 24 hours cost me? If that number is high, then redundancy is needed.

[JanethD]

Originally Posted by Serra

Well, if they have lost thousands in the few hours that these have been down, then I'd say they don't have a good disaster plan. They need to have redundant servers in different data centers. My customers with redundant servers haven't decided to switch to the backup servers yet, as this isn't their busy time, but I suspect that if they were losing thousands, they be on the backup servers already.

I've got clients that do thousands a day as well as clients that don't make a dime. Lucky for me I am not hosting the sites for the big clients but I do not know of any that leave their sites off line if they do not have to.

You can lose your rankings in the search engines and I know for a fact that one of my clients just had an order go through for $10,000.00. This is not a busy time for him but you never know what you are losing if your sites are not online.

[JanethD]

Originally Posted by Serra

It also helps for owner/resellers to know their own/customer's failure tolerance. How much will an outage of 12 or 24 hours cost me? If that number is high, then redundancy is needed.

I have no desire to be in the hosting business but I will say that after this everyone needs a backup server in place. But I would want one that would pick up the second my site went off line.

[Serra]

Originally Posted by JanethD

I have no desire to be in the hosting business but I will say that after this everyone needs a backup server in place. But I would want one that would pick up the second my site went off line.

If you use dyndns, the fail-over function will do that for you.

[GatorEricS]

As of 10:46pm, the following information is available:
At 4:55pm CDT, The Planet's Houston1 data center experienced an explosion and fire in their electrical room that knocked down three walls. There were no injuries, and no customer servers were damaged or lost.

As of 10:46pm, The Planet's engineers have just been allowed back into the building to physically inspect the damange. Early indications are that there was a short in a high-volume wire conduit. The fire department is prohibiting the activation of the back-up generator at this time.

This outage is impacting several HostGator dedicated server customers, and a few of our non-critical internal servers. All of our shared and reseller servers remain online, as do our monitoring machines. The Planet reports around 7,500 total customer affected, including HostGator.

Current assessment points to some service restored by mid-afternoon Sunday. We do not have any specific details when full service will be restored, at this time.

The Planet has their entire support staff, data center staff, engineering staff, and management staff called in, and they will be working around the clock to restore power, connectivity, and service.

We are sorry this incident has occurred, thankfully there was no loss of life, and should be minimal (if any) loss of data. Once again, your servers were not harmed by the event, they are merely offline due to lack of power. We will update this thread with more information as it becomes available.

[skeetr]

Definitely a very good thing noone was killed or injured.

Being an ex-electrician, its very easy to imagine how big an explosion from a short was. Its truly amazing the amount of power that can be expelled.

Its just too bad that the fire department isnt allowing the back up generator to operate, or none of the servers would be offline at this time.

Good luck to all those affected and I hope everything goes well with your customer sites as well as your own.

[GatorBrent]

Hostgator has 186 dedicated customers affected by this outage out of the total 9,000 servers offline. The datacenter believes sometime tomorrow morning / afternoon they will be able to get the second floor of h1 online. Luckily for us 90% of our affected customers are currently located on the second floor. So assuming their plan works most of you will be back online early tomorrow.

We are the datacenters largest customer and with that power I was able to pull a few strings........

The twenty or so customers that are on the first floor will be amongst the first boxes relocated upstairs. I wish I had a list of who is downstairs, but unfortunately I don't at this time.

I'd recommend whoever is affected by this outage to get some sleep because from everything I've been told it sounds like there's a .000001 chance you'll be up before daylight. I hate to say this but please don't call or put tickets in about this. The hostgator staff has been instructed to check this thread for all updates so if you do call they'll simply read whats here. (I'm updating the forum before staff)

Once everyone is back online we'll be issuing 100% credits for the month to those who are affected. While this is out of our control we are still going to stand fully behind our guarantee.
Thank you everyone.

[GatorBrent]

This thread is being locked please refer to the main one at http://forums.hostgator.com/showthread.php?t=33498