Major Outage 3/31/05

[GatorBrent]

As you already know your sites along with our site was off-line for hours today. The entire datacenter with over 20,000 servers was having problems and everyone was down. To make matters worse they did not give us any update until the problem was solved over three hours later.

Here's the official news provided by the planet...

"At approximately 4:15AM CST, a pair of redundant Powerware 500KVA UPS units failed creating a power failure in section B of our DLLSTX2 datacenter. Emergency teams were deployed within minutes and power was restored within minutes but intermittent power outages did occur until 6:45AM CST. Powerware, JT Packard, and electricians are currently onsite with over 100 Planet technicians working to resolve the issue. We do not anticipate any further outages . A formal RFO will be released once the team debriefs. We apologize for all issues that has caused. "

For those of you who do not understand we do not run the datacenter almost no web hosts in the world run their own datacenter, and if they do that usually means stay away.

The planet has one of the best reputations as far as datacenters go, and the network has been pretty flawless until the issues we all faced today.

We were taking phone calls during the problem however I'm sure you can imagine how many calls were coming in. We have a lot of cleaning up to do so please try to hold off from contacting us if at all possible.

I am extremely upset, and I understand your frustration!

Consider all the problems you had from this than magnify it to having 10k customers like us. Not a pretty picture! If they they release any more news on how this complete failure happened we will be sure to share it with you.

[amartins]

To begin theplanet needs to check the definition of redundant and redundancy

[GatorBrent]

Anywhere you go everything is always redundant until there is a problem. Our lives are one giant false sense of security.

[piperone]

is this a one off?,openheart surgery is not what i want,as a customer of less than one month I am worried,should I be or should i not be......million dollar question

[amartins]

Originally Posted by GatorBrent

Anywhere you go everything is always redundant until there is a problem. Our lives are one giant false sense of security.

Brent do u have any idea how incoming email would have been handled during this outage? Queued for further attempts at delivery or just immediatly undeliverable?

[amartins]

Hi,

Originally Posted by piperone

is this a one off?,openheart surgery is not what i want,as a customer of less than one month I am worried,should I be or should i not be......million dollar question

This is nothing more than my opinion but this was a pretty major thing and as far as those go they are *very* rare.

Last time I experienced such downtime was back in 2002(?) when the code red thing knocked down my previous host for 7 or 8 hours. Before that I have to think about a long downtime in 2000 (cihost fiasco).

Looks like years go by without this major of a prob happening taking out huge amounts of folks but thats just my experience.

2 hours into it I just faxed my customers informing them of conenctivity issues "we were working on" and had no ETA that cut down on the phonecalls we had but hey... thats only "doable" when your customer base is relatively low.

[M99]

Any estimates on when server 11 will be back online?

Frank

Edited: It's back up again.

[gregw]

Yes, things like this are very rare though no datacenter is ever "immune" from major problems like this.

A few years back we were co-located at a small data center in the Dallas area and a road work crew took a back hoe to the fiber line into the building. It was totally out of the datacenter's hands, but it happened. It took them 24 hours to get it back up.

My point is: thank goodness we have a great host who gives us the straight-up whenever things like this happen and don't try to cover it up. I'm glad we have someone who will ride them until it gets fixed.

[amartins]

http://forums.servermatrix.com/viewtopic.php?t=14363

Location: DLLSTX2

Severity: Level 5

Description: At 4:09AM CST, a power failure occurred in a redundant pair of Powerware UPS units feeding power to section B of the DLLSTX2 datacenter. The power failure was caused by a faulty fuse in UPS unit B-1. As the load transferred to UPS unit B-2, the spike in the load created an overloaded breaker and UPS B2 also lost power. This resulted in a power outage to the main power distribution unit feeding section B of the datacenter floor. Emergency teams were notified and the power was restored within 20 minutes. The power continued to cycle until 6:45AM due to the faulty fuse and the inability of the redundant UPS units to remain in bypass mode. Customers may have noticed several power cycles during this time period. Powerware, JT Packard, and electricians found the problem and replaced the faulty hardware and fuses. At 6:45AM, all electrical service was restored to normal and the NOC team began to bring all servers back online. The technical staff is currently placing a console on all servers to verify server restarts. Customers with operating systems that require a file check may have experienced extended downtime during the file check. Powerware and JT Packard will continue to monitor the UPS systems for the next 24 to 48 hours. The Planet does not anticipate any further outages.

[alikai]

Question: Does the 99% uptime guarantee apply to this outage?

[esl]

Question: Does the 99% uptime guarantee apply to this outage?

It should.

Note to HG members: I understand why you would want your money back because of the outage, but if your site is not used alot or is a personal website not used to bring in money, "put food on the table", you should not ask for a refund. I for one am not going to ask for a refund because I feel that this problem was not Hostgators fault and they should not have to pay me money for ThePlanet's error and ThePlanet's problem.

However, if you are a reseller and you feel that you lost money, request a refund, I don't blame you.

(I do not have alot of my clients on my HG account, I spread them out on different servers in different places in case this happens.)

[alikai]

Well, I'm not trying to place blame on Host Gator. Only thing is, Host Gator offers this guarantee and they "should" receive a refund of some kind from the Planet. At any rate, we "are" a reseller and we have several e-commerce sites that sell products everyday on their sites. We are responsible to our clients that their sites are up and operational. When the server goes down, especially for any kind of extend amount of time, that "always" impacts sales. If I have to refund hosting fees to my clients then I need to be reimbursed too, from somewhere, if possible. It's no more my company's fault then it is Gators. Gator should seek reimbursement from the Planet - that's what I think anyway. Finally, no one said we weren't resellers or attacked what I wrote, so I'm not trying to gripe or complain about being flamed - because that definitely isn't the situation. I just want to make myself and my feelings clear - that's all. And, I was curious if the guarantee applied here in this situation. Seems like if there's going to be a guarantee, then when "would" it apply if not at times when there are "real" outages like this one? And, of course, it's never really likely to be Gators fault, especially given that the servers aren't owned and operated by them. Anyway, just my own thoughts here.... I'm not trying to be a pain in the ass but a guarantee is a guarantee, I think, isn't it?

[JZ]

The uptime guarantee applies to both network and server failures.

"We

guarantee that your site will be up 99.9% of the time we host it. This guarantee includes both network and server failures. If your site isn't up 99.9% in any one month we will issue a full refund for the month."

- Jay

[amartins]

Obviously I don't know what SLA HG has with TP but apparently some TP customers are also protected on power faliures... naturally TP itself must have some sort of SLA with the companies who supply them with the electrical equip... so all up and down the food chain looks like lots of people are gonna be putting in for refunds.

At least now they must have learnt that electrical equip. degrades significantly with age so the upkeep may be better from now on.

[bheka]

So how do we proceed to claim a refund?

[amartins]

I guess that depends, maybe HG will come up with some plan sort of like what happened when the S hit the fan on that other server a few months ago.

To be honest I am so happy with the stability of my server over the last month that I am inclined not to bother. It helps that at least up till now none of my customers have blown a fuse over what happened. At most our only loss is in advertizing revenue.

Antonio

[iddgroup]

I asked in tech support, they said email sales@host...

so i did.

I wasnt happy about being woken up out of bed at 5am for a problem that wasnt mine. I lost one client because of it, and spent the whole morning answering calls, there is a cost for me to do that too.

I keep up my end of the agreement by paying my bill every month on time, i expect them to honor their guarantee as well.

what good is a guarantee if they dont honor it? thats like paying insurance, and when a hurricane hits, they dont want to pay for damage

[Amish]

A refund would be nice, but how much would it honestly be? I mean really now..

And I'd like to have an email saying "Hey Jim, something caught fire, burned down, and fell into the swamp" instead of wondering why my site (as well as HGs) are down.

[Stef]

Originally Posted by iddgroup

I keep up my end of the agreement by paying my bill every month on time, i expect them to honor their guarantee as well.

Originally Posted by Amish

A refund would be nice, but how much would it honestly be? I mean really now..

Whenever the uptime guarantee is not reached in a month, HG will refund you for the entire month. Only those who email in asking for it will be refunded.

Stef.

[Jme574]

so what is 99% of last month is hours? did this downtime make it fall below the 99% uptime guarantee?

[esl]

Yes it did. If you want a refund email sales@hostgator.com.

We guarantee that your site will be up 99.9% of the time we host it. This guarantee includes both network and server failures. If your site isn't up 99.9% in any one month we will issue a full refund for the month.

43.2min of downtime is the limit before refund.

60min x 24hr = 1440min
1440min x 30d = 43200min
43200min x .001 = 43.2min

.99 = 99%
.999 = 99.9

1.00 (100%)
- .999 (99.9%)
---------
.001 (.1%)

[Dawn]

Did every site go down? I'm on a semi dedi and didn't notice any downtime- not unless it was in the middle of the night here in the UK?

[amartins]

Hi,

Apparently not, a portion of the data center was caught up in the problem. But only HG can speak as to that...

You would definitly have noticed it. I am in the same timezone as the uk and we were gone between +/-11:00 and came back to life just past 16:15. Things came up gradually so some may have come back earlier and others later. At least judging buy what I've read on other forums (WHT and SM).

Antonio

[JZ]

Originally Posted by Dawn

Did every site go down? I'm on a semi dedi and didn't notice any downtime- not unless it was in the middle of the night here in the UK?

No, not all servers had a problem. My sites were, fortunately, unaffected.

- Jay

[esl]

How can you tell if your server was affacted?