Network Outage [ThePlanet/Houston1-DC]

This post is to advise that there is currently an outage in the Houston1 Datacenter.

More information can be found (officially) here:

http://forums.theplanet.com/index.php?showtopic=90185

My own personal server at Houston1 is down, and a technician has replied with:

"Houston Datacenter 1 is currently experiencing a critical electrical outage. Resolving this is issue is of utmost priority. We apologize for the inconvenience. Please standby for further updates. Thank you."


*update from brent*

Hostgator has 186 dedicated customers affected by this outage out of the total 9,000 servers offline. The datacenter believes sometime tomorrow morning / afternoon they will be able to get the second floor of h1 online. Luckily for us 90% of our affected customers are currently located on the second floor. So assuming their plan works most of you will be back online early tomorrow.

We are the datacenters largest customer and with that power I was able to pull a few strings........

The twenty or so customers that are on the first floor will be amongst the first boxes relocated upstairs. I wish I had a list of who is downstairs, but unfortunately I don't at this time.

I'd recommend whoever is affected by this outage to get some sleep because from everything I've been told it sounds like there's a .000001 chance you'll be up before daylight. I hate to say this but please don't call or put tickets in about this. The hostgator staff has been instructed to check this thread for all updates so if you do call they'll simply read whats here. (I'm updating the forum before staff)

Once everyone is back online we'll be issuing 100% credits for the month to those who are affected. While this is out of our control we are still going to stand fully behind our guarantee.
Thank you everyone.

[Sam]

OK, our Hstntx1 facility has been evacuated and we have emergency services at the site. I'm still trying to determine the exact reason and the scope of impact...

Why does my server have to be in DC1

Goodbye 326 day uptime.

I just hope my server boots up if the power has gone.

Where is that from, Sam?

[Sam]

Originally Posted by GatorZach

Where is that from, Sam?

The quote? http://forums.theplanet.com/index.ph...6&#entry592146

Oh, that's a different thread then. Didn't see it

[GvilleRick]

There was also a link to a post in webhostingtalk.com about the situation. That post is at http://www.webhostingtalk.com/showth...=697065&page=1.

[Sam]

UPDATE:

Today at approximately 5:45 p.m., a transformer in our H1 data center in Houston caught fire, thus requiring us to take down all generators as instructed by the fire department. All servers are down.

We are working with the fire department, with our facilities staff on site, to assess the situation.

To keep you updated, we will send messages every 15 minutes in Orbit and here in our forum.

I have a feeling this will be a long outage.

LOL @ Sam, you're quick, I was just about to post that.

I hope nobody got hurt. I will try to keep this thread updated, provided *it* doesn't catch on fire.

[GvilleRick]

I just copied it to post here as well. That's why Sam has over 3000 posts even though we signed up in the same month.

Official update from ThePlanet:

We have determined that no servers in the data center have been damaged. Nonetheless, they are down because power is out. Teams across the board are working to take appropriate action.

We will continue to keep you updated.

[Sam]

Update: ...

We have determined that no servers in the data center have been damaged. Nonetheless, they are down because power is out. Teams across the board are working to take appropriate action.

We will continue to keep you updated.

EDIT: You beat me Zach

[DarkSorrow]

blah.. So lame...

[GatorEricS]

At this time, the outage of Houston1-DC is affecting approximately 40 dedicated customers (of which we are aware), most of which are on our unmanaged dedicated plans*. This is not currently affecting any of our shared or reseller servers. Our monitoring and critical internal systems remain online at this time.

At approximately 5:45pm a transformer in the Houston1 datacenter caught fire, requiring evacuation of all personnel and requiring that all generators be turned off (as per the fire department). Our Datacenter's staff is currently coordinating with facilities management and the fire department to assess damage, co-ordinate plans, and restore service.

We will update with more information as it becomes available.

[GatorEricS]

Err, apologies, I forgot the asterisk from before.. Because I know people will ask: We do, from time to time offer unmanaged dedicated servers. There are currently none available. Access to these machines are handled through our sales team. Contact them for more information.

[TheExoticFishDOTcom]

Is this the server that IP 74.54.219.226 is running on ? All of a sudden my site went down with General Error 1040.

My database was deleted when I access my cPanel also ????

[GvilleRick]

Your site is giving an error indicating too many MySQL connections. The servers affected by this outage are all dedicated servers in the Houston DC. Your server is in the Dallas DC.

[Cranky]

I was scared when the server was going down. Glad it's being worked on.

[Sam]

Originally Posted by TheExoticFishDOTcom

Is this the server that IP 74.54.219.226 is running on ? All of a sudden my site went down with General Error 1040.

Nope, That error is due to mysql.

If you were on a server that is in Hstn1 then you wouldn't get to your site.

[TheExoticFishDOTcom]

Oh........

@TheExoticFishDOTcom, Please post the ticket ID of your support request for this in this thread.

[Sam]

Update: ...

In our latest assessment, we have determined that networking gear has not been damaged, but we are without power so assessments continue. All disaster recovery systems are in motion, and we have teams already working in the data center.

[Serra]

My server is down too. Looks like the generator fire didn't hurt the data center, which is good news. Just the electrical.

With any luck we will be back up in a few hours...

Update: Looks like they are working in the data center now, so they will be ready when the power is restored. Not as bad as it could have been.


Edit: I was worried it was a billing issue... logged into Orbit and my server wasn't listed... about freaked....

[Sam]

Originally Posted by Serra

With any luck we will be back up in a few hours...

I hope you're correct and GatorVincent is wrong:

Originally Posted by GatorVincent

We have talked to the DC at this time, the ETA to be back online is tomorrow morning. If this changes, we will keep you updated. There is really nothing we can do but wait, sorry folks, I know this is frustrating for some. Thanks for your patience everyone.

[pbradish]

Not good. I have clients who are losing thousands because of this and are already requesting new server hosts.

[Serra]

Originally Posted by pbradish

Not good. I have clients who are losing thousands because of this and are already requesting new server hosts.

Well, if they have lost thousands in the few hours that these have been down, then I'd say they don't have a good disaster plan. They need to have redundant servers in different data centers. My customers with redundant servers haven't decided to switch to the backup servers yet, as this isn't their busy time, but I suspect that if they were losing thousands, they be on the backup servers already.