May 12, 2009 Network Outage Details

[GatorDHanna]

Update on May 13, 2009: We noticed approximately 10 minutes of downtime for a few of our dedicated server customers who have dedicated servers in the H1 and H2 datacenters located in Houston. Sites were unavailable for approximately 10 minutes, starting at 10:20 AM CT. This second outage did not affect shared or reseller customers whatsoever. We are awaiting further details from The Planet about the cause of this outage. From what we know so far, the issues in Houston outage were on the CAR (customer access router) level and not related to an IP block or yesterday's issues.


On May 12, 2009 around approximately 4:50 PM CT, a number of HostGator's servers went down due to a widespread network outage at The Planet, HostGator's datacenter and network provider.

The Planet has provided the following information, which we will continue to update as it becomes available:

• Issues limited to The Planet's H1, H2, and D6 datacenters.
• There was a block of IP addresses that were not advertising correctly to the rest of the Internet.
• The Planet is investigating

Approximately 25 minutes later, service was fully restored to HostGator's customers and to HostGator's main website. Support channels (phone, live chat, tickets) continue to be busy, but response times should return back to normal soon.

During the brief outage, HostGator provided frequent updates on its Twitter account available at http://twitter.com/hostgator.

The Planet is investigating the cause of this issue and we'll report back what we hear from them on this forum post and on Twitter.

If your website was down more than 45 minutes due to this outage and any other downtime events in the past month, please let our sales team know. We will of course honor our standard 99.9% uptime guarantee.

Thank you for your patience and understanding.

[jsmith43]

Thank you for the timely update. You may note that the phone lines were down during this time too, making it impossible to call in to figure out what was happening.

[RainbowViper]

Gator 396, at least, is still "hiccuping" badly. Although I can hit my site, cPanel is taking over 2 minutes to load. It eventually does, but during that time I'm seeing LONG waits while the status bar is saying "connecting to www.google.com". It's probably related, just posting here for FYI.

[Revo]

What about emails sent during the outage? Are they lost? I have emails that I'm expecting and haven't received them yet.

[RainbowViper]

They *shouldn't* be lost, no. Just delayed.

[fanfavorite]

Originally Posted by jsmith43

Thank you for the timely update. You may note that the phone lines were down during this time too, making it impossible to call in to figure out what was happening.

Locally a few shops have done the old pack up and leave, so I was worried that maybe Hostgator had done the same since all sites were down and phone numbers weren't in service during that time.

[Chimpie]

Thank you, HG, for keeping us informed via Twitter.

[quietFinn]

Originally Posted by Revo

What about emails sent during the outage? Are they lost? I have emails that I'm expecting and haven't received them yet.

They are not lost, the sending server puts them in queue and sends later.

[fanfavorite]

Yeah lucky I searched and found the twitter site, otherwise there may have been a few holes in my office walls lol.

[GatorJacob]

Originally Posted by RainbowViper

Gator 396, at least, is still "hiccuping" badly. Although I can hit my site, cPanel is taking over 2 minutes to load. It eventually does, but during that time I'm seeing LONG waits while the status bar is saying "connecting to www.google.com". It's probably related, just posting here for FYI.

I'm so sorry. I was able to get in fine with Root WHM, with no lags / slow downs. Are you still experiencing problems connecting to your WHM? If so, please feel free to PM me, or open a support ticket.

[LoonyPandora]

Originally Posted by Chimpie

Thank you, HG, for keeping us informed via Twitter.

Same from me

[fanfavorite]

Originally Posted by scribblesteve

99.99% guarantee. I can understand how accidents happen, but where does the guarantee come in? So if it goes down again anytime the rest of the year do I get refunded?

Form Hostgator site:

"If your site isn't up 99.9% in any one month we will issue a full refund for the month."

So from above they say 45 minutes. I guess that is the line for a given month.

[quietFinn]

Originally Posted by scribblesteve

99.99% guarantee. I can understand how accidents happen, but where does the guarantee come in? So if it goes down again anytime the rest of the year do I get refunded?

"We guarantee that your site will be up 99.9% of the time we host it. This guarantee includes both network and server failures. If your site isn't up 99.9% in any one month we will issue a full refund for the month."

[Xtreme2damax]

I was a bit worried, everything was working fine when I was booted into my install of Windows 7 RC, then I restarted and booted into Vista only to find that my website wasn't loading.

I was also unable to load the Hostgator website to fill out a ticket, and figured it was a widespread issue.

I'm relieved to know the actual cause of the problem, thanks for updating us.

I was scared, even though my last database backup was this morning, my last backup of my files was over a week or two ago.

[Chimpie]

According to my very rough math, your server will need to be down 45 minutes to get a refund. According to the post above it was only down about 25 minutes.

I, for one, feel that today's outage was not a HG issue, therefore, will not be seeking any kind of refund.

[regentronique]

The 4 servers i tested were down for a full 40 minutes...

[GatorNathon]

Originally Posted by Xtreme2damax

I was scared, even though my last database backup was this morning, my last backup of my files was over a week or two ago.

Just to make sure everyone is aware there was no chance of data loss during the outage. The servers were all powered on and waiting to do their jobs but the network connection to them was down.

So there was not so much as a power failure to have the possibility of data loss.

[RainbowViper]

Nathon, I'm Shared Hosting. Yes, cPanel is still talking forever to load for me, and I *think* because it's waiting to connect to www.google.com as stated before.

[GatorNathon]

Originally Posted by RainbowViper

Nathon, I'm Shared Hosting. Yes, cPanel is still talking forever to load for me, and I *think* because it's waiting to connect to www.google.com as stated before.

I'm able to get into your cpanel with no issues, I didn't even see it try to load anything from google.com. This is going through root WHM to your account though so maybe something is different. I went ahead and restarted cpanel on your server just to be sure.

Can you go ahead and check this for me now and let me know if the problem is corrected. If not go ahead and PM me as I don't believe this issue would be related.

[jmeadows]

Maybe having all hostgator services on the same IP range/data center is not the best idea?

Without Twitter (thank you for that), none of us would have know what was up (with the IP phone system down as well as the support chat).

[slapshotw]

Originally Posted by RainbowViper

Nathon, I'm Shared Hosting. Yes, cPanel is still talking forever to load for me, and I *think* because it's waiting to connect to www.google.com as stated before.

In a completely unrelated problem, all/most Google services are completely down for many people around the internet. See here:

http://search.twitter.com/search?q=google

It may or may not have to do with OpenDNS.

So if you have things like adsense, Google-hosted jQuery, etc... this can cause problems for *you* if you can't see Google, but not for people who aren't having Google issues.

[jz1977]

i guess my server is getting a full refund. its been down over an hour. http://jeffzakovec.com
regards,
-Jeff Zakovec

[GatorNathon]

Originally Posted by jz1977

i guess my server is getting a full refund. its been down over an hour. http://jeffzakovec.com
regards,
-Jeff Zakovec

That server is currently up and accepting connections as normal. I can also confirm this off of the server's access logs. If you are still unable to see your site on the server this would be an unrelated problem.

[alvin]

I own the website http://www.files2send.com/ and actually have a backup server prepared for situations such as this when the main server goes down. However, with hostgator DNS servers and DNS manager service down as well, my backup server was useless. Is there a way that you can at least establish some redundancy for your management servers so that we can have a way to see the nature of the problem and/or redirect our servers to another site via the DNS? I too like an earlier poster was concerned about HostGator doing the pack-and-run until I discovered by pure luck the twitter account. My site being down was troubling enough, having the provider's web sites down made me downright paranoid, but not having the DNS for any of these sites work just plain and simple freaked me out.

Meseret

[bluemirrow]

Due to Network Outage my websites are still down.
according the live chat (ChatID: 488938)
"Yes we are currently having a problem with that server and are working on it right now."
I hope they will resolve the problem my websites are down now more than 2 hours.I believe I should get a full refund for the month