May 12, 2009 Network Outage Details

[marko1]

Is the twitter site the "official" place to get updates during these outages? I know it is probably useless to phone or email if it is system wide but being a new customer I didn't know where to go to get a status update. It would probably be more professional to have this forum hosted elsewhere or have some sort of redundancy as this was the first place I tried to check as I am sure many others did the same.

[GatorNathon]

Originally Posted by bluemirrow

Due to Network Outage my websites are still down.
according the live chat (ChatID: 488938)
"Yes we are currently having a problem with that server and are working on it right now."
I hope they will resolve the problem my websites are down now more than 2 hours.I believe I should get a full refund for the month

I apologize for the confusion that this was not made clear in you chat but this is unrelated to the issues earlier today. There is currently an attack being mitigated on that server.

This has nothing to do with the outage from earlier today and ALL issues from that are now resolved.

As per your chat since you have experienced greater then the 45 minutes of downtime due to both the outage earlier and the current attack on this server you would be eligible for an SLA request. You would need to email sales@hostgator.com as requested in your chat.

[airjrdn]

This is what InternetSeer emailed me about the downtime:

Recovery Alert

Your Web Site is no longer on error

URL: http://www.themintonfamily.com
Time: Tue May 12 18:35:44 EDT 2009
Total Time on Error: 45 minutes, 24 seconds

[GatorNathon]

Originally Posted by marko1

Is the twitter site the "official" place to get updates during these outages? I know it is probably useless to phone or email if it is system wide but being a new customer I didn't know where to go to get a status update. It would probably be more professional to have this forum hosted elsewhere or have some sort of redundancy as this was the first place I tried to check as I am sure many others did the same.

In this instance today, during the 25 minutes that we were unable to communicate this on our own forums, twitter was the only location where we could post system status notifications.

Doug was luckily able to keep everyone in the loop on hostgator's official twitter account and let everyone know what is going on. I know that this might not be ideal and have suggested that we keep in off network status page in case of emergencies.

I have thought about this option but due to the rare nature of these kinds of large outages it would seem pointless to keep an off network server ready at all times. In reality this was just a short disturbance of service and if it would have taken longer we would have implemented steps to get our main site and forums online under any means necessary.

So the unofficial-official answer is yes, status updates are available via twitter but we will always keep our main priority at the forums and the Houston support department.

[GatorNathon]

Originally Posted by airjrdn

This is what InternetSeer emailed me about the downtime:

These are usually not 100% accurate because they only check your site in intervals and not continuously. If you do feel that you are owed a credit please feel free to email sales@hostgator.com and we will look into this for you.

[HalbrookTech]

Originally Posted by airjrdn

This is what InternetSeer emailed me about the downtime:

How often does it check your sites?

[regentronique]

Is it possible to know from what source you got your 25 minutes of outage?

Because from my logs analysis i just founded that superbird server was down for 43 minutes...

HostGator forum was down for 40 minutes also, i was checking each minute...

[GatorNathon]

Originally Posted by regentronique

Is it possible to know from what source you got your 25 minutes of outage?

Because from my logs analysis i just founded that superbird server was down for 43 minutes...

HostGator forum was down for 40 minutes also, i was checking each minute...

This is going off of actual employees connectivity and also from the data center's logs. If you believe your server was down for over 45 minutes feel free to email sales@hostgator.com and we will look into this issue for you.

[rec9140]

Originally Posted by GatorNathon

Doug was luckily able to keep everyone in the loop on hostgator's official twitter account and let everyone know what is going on. I know that this might not be ideal and have suggested that we keep in off network status page in case of emergencies.

I have to agree with the original poster, that having a system that is hosted outside of your network, servers etc. would be better.

Why? For one not every one can access the "service" you used. Specifically I block that as its a time wasting system that my employees have no need to access at work. Additionally I don't use it for any reason. Theres a better way to communicate with your customers,

I want to go to a web site and get info that is regularly updated with timely and accurate information. Thankfully this was not a bigger or longer issue.

Had it been well..... lets just not even think about it.

Better communication with systems setup outside HG's DC's, network etc. is a must.

[GatorNathon]

Originally Posted by rec9140

Better communication with systems setup outside HG's DC's, network etc. is a must.

As I had stated in a previous post we will be looking into something like this.

Please remember that this was a minor amount of time and if something major were to have happened we could easily redirect traffic for forums.hostgator.com or something like that to an IP of another data center.

Why wouldn't we just do that immediately for the amount of time it took for today? Simple answer is that by the time the DNS migration and setup of the page showing the status was complete the original network connectivity would have been restored.

As we use the planet as our data center provider we have access to many locations inside of TX but multiple data centers were effected by this problem.

[magzone]

A massive heads up to hostgator for using twitter to keep us informed, the only page i could get on to with hostgator was this one https://secure.hostgator.com/support.shtml nothing else worked it might be an idea to drop a link to twitter in there, well done whoever thought of it. Wasn't bothered too much about the downtime it was during the night here at least we were kept informed

[GatorJacob]

Dear Customers,

At this time we are experiencing another outage in our data centers. Please be patient, and we will update this thread as soon as we have new information regarding this outage. I apologize greatly for the inconvenience, We are aware of the issue and working to address this for you as quickly as possible. We thank you in the meantime for your much appreciated patience while we work to resolve the issue at hand.

[GatorJacob]

Dear Customers,

We truly apologize for the inconveniences and the frustrations that this outage has caused. We are seeing the servers coming back online and getting connected properly.

If you are still seeing problems, please contact our Live Chat for help. Once we know the cause of the outage, we will update this thread. It looks like a total of 10 minutes of downtime that we saw from the office - 10:20 - 10:30.

Again, we apologize and we thank you for your patience during this time.

[wwobn]

So what actually happened/is happening? For those of us who are intelligent enough to understand.

[GatorJacob]

Brad,

We are still waiting to hear from our data center on what the issue was this morning.

Once we know, we'll post an update in this thread.

Again, I'm so sorry for the problems.

[ldearing]

Originally Posted by alvin

I...have a backup server prepared for situations such as this... However, with hostgator DNS servers and DNS manager service down as well, my backup server was useless...

for a very modest amount i use a third party vendor for my enterprise DNS as well as backup sites at yet another provider and "store and forward" backup email.

@Gators: Once again, thank you for your updates!

[GatorDHanna]

I've added some additional details about the brief outage we saw today to the original post in this thread. If you have any other questions, feel free to let us know.

[Shopgeek]

Thanks for keeping us updated. Using Twitter was a great idea.

My other webhost would have blamed me and given me the digital finger for daring to question their omnipotence.

[granviller]

OMG, twitter is down. where do we do if the forums and our servers go down again.



when everything went down yesterday evening, I went crazy trying to figure out how to get a hold of HG. Phones were down, website was down, and the weird thing is that I was able to connect to one of the dedicated servers we have and then ssh into the one we have in Houston, so I knew the server wasn't the problem, but that it was a connectivity issue.

I ended up on The Planet's forums and they didn't have anything either at the time, but even their phone number gave me a fast busy signal. People over there are getting crazy mad about getting answers. All I wanted to do was to let someone at HG know, Houston - we have have a problem. .

We've been with HG for over a year now and have had only one time that we needed a complete reboot. I find that HG and The Planet provide great service and fantastic up time, and I'm understanding enough that something things just happens and all hell breaks loose.

Anyways, I just wanted to let you guys know that HG and TP are doing a great job. I just hope nothing happens while twitter is also down.

granville

[quietFinn]

Originally Posted by granviller

I ended up on The Planet's forums and they didn't have anything either at the time,...

There was a thread, but it was in "Suggestions/Comments" forum area

[granviller]

Originally Posted by quietFinn

There was a thread, but it was in "Suggestions/Comments" forum area

yeah I found it, but at the time I was looking it was brand new and they didn't have any real status as to what was going on.

I was at work when it went down as it was just before 5pm, so I caught the disconnect in the act. Needless to say it was panic mode when I couldn't pull up HG's website. I even called a friend who lives in Houston and asked if anything big had happened. He said he didn't know of anything.

Granville

[GatorDHanna]

Hi all,

Just as an update, we have yet to hear anything of value from The Planet about the cause of the larger issue on Tuesday and the minor one on Wednesday.

Publicly, they are staying pretty quiet at the moment (the first incident was caused by "human error" and we're told the second incident was unrelated), but as we hear more, we're going to provide updates.

We are also looking into mirroring our site with a datacenter besides The Planet. At the moment, our main site is redundant on several levels (it rarely goes down), but all of the levels are eventually dependent on The Planet's network. We're looking into changing that so we can ensure our site stays up in the event of another major network outage at The Planet.

I'll keep this thread updated as I hear more. Thanks for your patience and understanding.

[garyopa]

Don't worry "Gator" techs, Google had an "outage" today also,
with it being blamed to "internal error" that routed a percent
of their traffic via "asia".

Guess the internet needs a complete backup of itself. Google included.

http://bits.blogs.nytimes.com/2009/0...s-services/?hp

[jz1977]

Hello,

My domains will not work since that morning of the outage.

Jeffzakovec.com
nweststorage.com
spitonthis.com

i know the sites are online because i checked them from a serverbeach server.

my question is... why are these still offline in lincoln, nebraska?

any dns changes? cuz non were made on my end.

i'm looking for answers, because im sorta stressing over this.

even a tracert is dead

[quietFinn]

Originally Posted by jz1977

Hello,

My domains will not work since that morning of the outage.

Jeffzakovec.com
nweststorage.com
spitonthis.com

Works well from here.