OUTAGE: mercedes (67.19.161.34)

mercedes.websitewelcome.com (67.19.161.34) is currently down, as it is having some problems with its hard disks.

We're working with our dedicated technicians at ThePlanet to swap these servers with brand new ones. Once this is done, all customers on these servers will be upgraded to the new packages listed here: http://www.hostgator.com/resellers.shtml

We apologize for any inconvenience caused by this. Rest assured that we are working as quickly as possible on getting this issue resolved. We will continue to keep you updated on our progress using this topic.

[dmcl1]

Thanks for at least posting here. Unfortunately I just signed on a new client yesterday. Will appreciate the space and bandwidth upgrades and pass it on to the clients.

[vizualeyez]

Is there any updated information.

[t_alan]

How long have they been down, and how long are they expected to be down? Also - aren't those ALREADY the reseller plans? ( http://www.hostgator.com/resellers.shtml )

[dmcl1]

They've been down since at least 7:30am EDST.

For those of us with older reseller plans, it will be an upgrade.

[vizualeyez]

I am showing down since about 3:30am EST. That is what time outlook quit receiving files. Going on 10 hours now, and no updates?

[Serra]

Originally Posted by vizualeyez

I am showing down since about 3:30am EST. That is what time outlook quit receiving files. Going on 10 hours now, and no updates?

They are swapping hard drives, don't expect any kind of resolution for at least 12 hours. Don't expect any updates either. Its difficult if not impossible to give estimates on how long the process will take. They will know when it will finish, when it is finished.

[GatorBrent]

I apologize everyone I didn't get into work until around 11 today, and I stepped in the door to be notified of the problem with the mercedes server. Our best admin Dave Collins has been handling this since the box went down, and in all honesty me knowing about it or not wouldn't have changed anything other then me being able to keep you up to date. So far communication has been pathetic, and for this I apologize.

So here's what's going on....

Yesterday the server had to be rebooted for the first time in months. We checked everything out after reboot and didn't see any hardware issues or signs of a failing drive. It's common for a user to overload a box and crash a server so we checked what we could and didn't think much of it.

At 7amish today the server crashed with almost no load and no memory being used. It failed to boot up after it kicked one of the drives from the array. We tried getting the kicked drive to work, but we had no luck with that drive. The existing drive is currently in such poor shape any service running causes the box to go offline.


Dave managed to recover mysql, and some other important files off the box. We are currently restoring the box from backups and this could be as quick as 5 hours, or as long as 12+ hours. We'll have a better idea once the restore is a little more into the process as it's all automated. Right now we believe it will be closer to the 5 hour mark.

Since the box went down we've been tar balling backups so the restore process was started right away.

The bad news....
More downtime until restore completes. eta 5-12 hours. (don't hold me to it)
Home files will be out of date by about 4 days.

The good news....
The server will be on a new top of the line conroe server.
We will be upgrading everyone to the new plans.
Mysql will be up to date as we were able to recover that partition off the bad drive.
We believe we can recover mail so it will also be up to date. I'd say this is still 50/50 worst case mail will be missing anything that was downloaded in the last few days.

This is as much as I know so far, and I'll update everyone as I get more news.

[dmcl1]

Brent,
Thanks VERY much for the full update.

[GatorBrent]

Everyones sites should be up to a certain degree at the moment. We have apache running on the old bad server / drive for the time being while we finish a few things on this restore. So during this time it's going to be unstable going up and down. Contacting us won't speed it up.


We now believe the restore will actually be done much quicker then the 5 hours I was saying earlier. It could be done in 3-4 from what were currently seeing.

[davkrev]

What will happen to emails being sent to domains hosted on Mercedes during this whole downtime process? Will they be bounced back or will they eventually be delivered when everything is back to normal?

[Serra]

Emails will either not be sent, the sender gets a warning that the email can not be sent or more likely, they will be held in the sending system's queue until the server can be contact. Those emails will arrive based on how long they have been held. Older email will arrive later, newer email will arrive faster.

In cases like this, only email actually on the system could be lost. Those in transit normally show up at some point.

If your system has a backup MX record, you shouldn't have any problems.

[t_alan]

HTML sites are up, but my Drupal sites are giving a "Unable to connect to database server" message. Any idea when MySQL might be up and running again? Will we need to do anything to reconfigure the Drupal installations?

[ThreeD]

I guess my concern is the data backup. How much data will be lost, or when was the last complete backup run on the server?

I can see my sites are up and running, but they all give me database errors

EDIT: At 5.25pm EST all my sites are down again

[century]

Seems strange that two other servers have had 'hard disk' problems

My customer has legal documents in the quey that can be very costly for him. Anybody know what can be done to retrieve it????????

[slapshotw]

This is why I keep backup mx records on my important email domains. Even if the server goes down, I know all the email will come through as soon as it's back up with no delayed delivery or bounce messages.

[t_alan]

looks like mysql is back up. So everything seems to be working now except email.

by the way - how does one "keep backup mx records" and what's the process for retrieving those messages?

well... email is downloading - no -- wait -- servers are up, but no messages downloading

[GatorBrent]

The restore is completely done. We are now just doing little fixes here and there to get it where it needs to be.


Mysql was completely up to date.

Everything else such as home and mail is going to be from May 6th. We weren't able to recover anything else of the bad drives.



We will be increasing everyone to the new plans shortly. We thank you for your patience and look forward to once again providing you all with reliable hosting.

[GatorBrent]

Century we have over 2,000 servers now. It's pretty rare now a days to not have a hard drive failing on a box.

The last few days has been more then normal going about same time so you are right there. Now were just down to chrysler and caliber doing chassi swaps that currently have bad drives. The difference is they didn't have downtimes like mercedes did while we rsync to replace.

[ThreeD]

first off, great job for putting everything back together.

but here is something I don't understand. If HG takes complete backups of their servers every sunday, how come all my sites/files that have been reuploaded are dated back to March 2nd, 2007?

[davkrev]

Weird... don't think this should be happening. I went to modify a password on one of my users accounts and got this output in the process:

Updating ftp passwords for all users
Warning: Unable to determine drew007's main domain. Skipping vhosts password update for this user.
Warning: Unable to determine rhodesta's main domain. Skipping vhosts password update for this user.
Warning: Unable to determine hospedaj's main domain. Skipping vhosts password update for this user.
Warning: Unable to determine yumafree's main domain. Skipping vhosts password update for this user.
Warning: Unable to determine primafin's main domain. Skipping vhosts password update for this user.
Warning: Unable to determine primapho's main domain. Skipping vhosts password update for this user.
Ftp password files updated.

However, the password was modified okay and they can get in to their account fine. I don't know what that other output is though? Any ideas?
Ftp vhost passwords synced


----

ALSO -- Horde (webmail) gives some php errors when trying to access it. Anyone else experiencing this?

[slapshotw]

Originally Posted by t_alan

by the way - how does one "keep backup mx records" and what's the process for retrieving those messages?

The messages are automatically delivered to the main server by the backup servers once the main server is back up and working. PM me and I'll help you set them up, this isn't really the right thread for it.

[dmcl1]

Originally Posted by GatorBrent

Everything else such as home and mail is going to be from May 6th.

You sure about that date Brent? I just checked on a client who has been in the process of adding tons of products to her store over the last few months. Looks like the data side of everything is okay, but we've lost all the images for these products. Files that are there now look to be from MARCH.

Haven't checked yet but I suspect that means the entire shipping module I wrote and added last month isn't there either.


Another client's straight HTML sites were fine at 3, but now deliver 404 errors. Can't FTP.

[BGE]

Originally Posted by slapshotw

The messages are automatically delivered to the main server by the backup servers once the main server is back up and working. PM me and I'll help you set them up, this isn't really the right thread for it.

I am really surpised at this. All except one of my websites are up but two of them have old pages. I don't understand at all.

How could the backup not take the most recent files?

[ThreeD]

Originally Posted by dmcl1

Files that are there now look to be from MARCH.

I have the same problem, all my sites, files are dated MARCH two months lost on several sites, there have to be more recent updates available!!