City Update

Unfortunately the server city.websitewelcome.com had issues with the Raid Controller a few moments ago and is currently offline while we replace the raid controller. We expect to have the server back online within the next 15-20 minutes. On a good note, there was no data loss during this issue.

We appreciate your business and thank you for your patience.

Have a wonderful day!


>>>>>>>>>>>>>>>>>>>>>>>

Read bottom post for updates

>>>>>>>>>>>>>>>>>>>>>>>>

[GatorBrent]

They are currently running a FSCK, and have been for a while. it appears the failing hardware caused some corruption. we will know more when the fsck completes any minute.

[Fixprod]

what happens with received mails during this OFF time ?

thx

[quietFinn]

Originally Posted by Fixprod

what happens with received mails during this OFF time ?

thx

They will be queued in the sending server(s) and delivered when the box is back up

[GatorBrent]

Nobody is really sure what happened but here are the facts............


1. the server went down
2. it gave a raid error
3. We had to run a fsck which took about an hour. (to fix corruption)
4. The box still would not come online so we had the data center do knoptix to get us ssh.

5. Reviewing the data everything was corrupted and non-Salvageable.

We are going to have to restore from our backups that ran on Sunday. There will be a few days of data loss.

The backups will be restored from A-Z with A being first. The closer your username is to "a" the sooner you will be online. There are about 55 gigs of data needing to be restored which I'm guessing will take about eight hours to fully restore.


Unfortunately something like this is impossible to prevent, but since we are unsure what piece of hardware caused the corruption we are having a complete chassi swap. The ram, motherboard, tower, cables, drives, 3ware, everything is being replaced.

[Fixprod]

Originally Posted by GatorBrent

The backups will be restored from A-Z with A being first. The closer your username is to "a" the sooner you will be online.

what "name" do you reffer to ? real name or main account name ?

thx

[dutcheagle]

When I send a mail to one of my domains it bounces so there is no queue.

[dutcheagle]

What about your statement;

Every server in out fleet has a Raid1 setup that provides instant backups. ????????????????

As a raid 1 is a mirror why are you calling this a backup???
corruption on one disk is replicated to the second disk. This is NOT a backup it only prevents data-lost in case of a disk failure.

Are you telling me that only once a week a backup is running. If so please take other measurements or at least tell people so we can do our own backup more often.

How do we get our refund as this downtime is already less then 99.9% of a month uptime.

I'm very concernd about the whole situation.

regards
KS

[Fixprod]

any update on the recovry process ??

if email bounces thats really bad !!! thats many ppls email not comin through !! really bad !!

[GatorBrent]

It is going by a site's username and running very fast. I'm hoping it will be completed in 4 hours. Sites are coming online one by one.


dutcheagle your right it isn't a backup to corruption from bad hardware. No one said it was. It's great for failing drives.

"Are you telling me that only once a week a backup is running. If so please take other measurements or at least tell people so we can do our own backup more often."

Backup storage is not cheap we spend $5k+ a month on backup servers alone with our current method.

Not to mention when backups run the servers go substantially slower causing intermittent service outages, and slower server response times. We could provide you with whatever backup solution you want it's just going to cost more money and would require you to leave a shared environment.
Once the server is fully restored simply email sales for a full refund for the month. Out of the 3+ years in business and 200+ shared/ reseller boxes we've only had this type of hardware failure a few times. We do everything we can to prevent something like this and are very sorry for any time of server outage or data loss.

[dutcheagle]

Originally Posted by GatorBrent

dutcheagle your right it isn't a backup to corruption from bad hardware. No one said it was. It's great for failing drives.

Thanks for your explanation.

About the raid1 statement as a backup please read the info under the questionmark next to Instant Backups.

[GatorBrent]

It's about 20% done now. We will be watching it until it's done.

[Jme574]

Brent - I am not on City but I would like to thank you for the work you guys have done. It seems like you have taken many steps since the Supra incedent. Changing out the chasis now instead of putting it all back together then having it fail again. That is great! And the fact that they are getting updates regularly here about their server, they should be happy.

It could be worse for them. They should be glad they didn't go through the Supra problems. Still with you guys! Keep up the good work!

[novatempus]

Hello,


I'm a reseller of the server affected by the problem.

How is at the moment the state of the issue?

I suppose that this fall is within the the "99.9% uptime" terms. Can I reclaim any compensation for it?

I hope that the technicians can finish it as rapidly as possible.


Greetings.

[etouffee]

I've been down for 12+ hours now, and that's really pushing the limits of acceptable service from a hosting provider. I do believe there should be some sort of refund or compensation for this.

In any case, what's the latest estimate for being back up?

Hi,

We do apologize for the amount of time the restore is taking but please understand that I personally have not slept since 1PM in the afternoon yesterday and my shift ends at 2AM. The staff at HostGator is dedicated to our customers and we do whatever it takes to get the job done. Most hosting companies don't even keep backups of their servers but we stick to our word at HostGator. I have been working on City since 5PM EST April 5, 2006 and do not plan on resting until the server is completely restored.

A credit of one month will be provided to the users on City as we also standby our Uptime Guarantee. Please try to understand that this was a hardware issue unrelated to HostGator's service as a corrupted file system can happen on any server at any company but fortunately we're always prepared for unfortunate events such as this one.

City is currently half-way through the restore and we expect it to be completed within the next few hours. Once again, we appreciate your business and thank you for your patience. If there is anything we can do to assist you please do not hesitate to ask!

[dutcheagle]

I appreciate your effort nevertheless it is an unpleasant experience and needs to be looked at in a later time to prevent this much of data lost in the future.

Please let us know when the restore jobs are finished so we can verify our websites and inform the users.

[etouffee]

Yes, we all appreciate your efforts to address the problem, and I'm not angry with anyone on a personal level, so please don't take my comments that way. Still, we're at almost 17 hours down now for one of my sites. All my sites are back up except one, which does not show up in "list accounts" in my reseller cpanel. This scares me; I hope it is not completely lost. Are there still accounts that have not been restored?

Hi,

Yes we are still in the process of running the restore. I will be sure to keep you all updated as soon as it's completed. I deeply apologize for any inconvenience this has caused you all, and greatly appreciate your patience and understanding.

[Fixprod]

It seems im back up but my clients are not... this is normal ?? ( reseller hosting plan )

thanks..

[prevail]

A 1 month credit hardly compensates for the loss of sales that this outage has caused.

Just to reiterate, the accounts are being restored one at a time in alphabetical order. This is why some of the accounts are showing up, the ones that are not showing up are still being restored (or will be within time).

[etouffee]

Originally Posted by GatorJesse

Just to reiterate, the accounts are being restored one at a time in alphabetical order. This is why some of the accounts are showing up, the ones that are not showing up are still being restored (or will be within time).

Ok, so can you give us a hint or something? What letter are you working on now?

[prevail]

They are obviously on the first half of the alphabet as we are still off at "n"

We are currently on M but have already copied over 70% of the data so it shouldn't be too much longer.

Thank you all so much for your patience!