Emergency Maintenance : brougham

[GatorJustinM]

Brougham has been taken offline for emergency maintenance. At this time, we do not have an estimate as to when the server will be back online. However, we are working as fast as we can to restore service. Please direct any questions to support@hostgator.com.

[GatorJustinM]

There is still a fsck running on the /home partition which is taking a while to complete. The current progress is at 60% and we will update the forum when this is done. There is also a failed drive which will need to be replaced thereafter. We appreciate your continued patience during this time.

[GatorNate]

The fsck finished enough that we could boot the server and get most of the drives mounted. We are having to do a second pass on /home so right now all site data is not able to be served, however email and dns are back online so any email sent to your domains will queue for delivery as soon as /home is added back in.

I have put up a unbranded maintinance page so that all sites show:

Under Maintenance :: Thank you for your patience!

We will update this thread as the work continues.

[GatorNate]

The FSCK of brougham continues, it is on its third full pass of every single inode cleaning for inodes with multiply-claimed blocks. This is the last major pass, but when its gets this low level, it stops providing specific percentages of compleation, so we can't give an ETA at this time.

[GatorNate]

The FSCK is continuing, it had to restart again to adjust after finding some issues, but it is at 80% finished now. We will continue to update this thread as this progresses.

[GatorNate]

The FSCK continues, it has found a spell of very bad inodes and has not given a percentage in a while. We are continuing to watch it and will get the sites back online as soon as we can.

[GatorMark]

The FSCK is still running at this time. Unfortunately we still do not have an ETA on when this will be completed. Thank you for your continued patience.

[GatorMark]

The server is currently up with the /home partition mounted as read-only while we move data to a new server and replace any corrupted/missing data from backups. For the most part the corrupted data is public_html directories. Sites that have intact data should be loading at this time. All others won't be up until the backup data is copied to the new server and it's brought online.

We'll update this thread with any new information as it becomes available. Thank you for your patience.

[zar2]

What about the email service? Can users on the domains send and receive email?

[GatorMark]

Originally Posted by zar2

What about the email service? Can users on the domains send and receive email?

The /home partition is mounted read-only and no data can be written to it. That means you can check email that was previously delivered, provided that data is intact, but you cannot receive anything new at this time. Outgoing email should be working but the sent messages won't be stored on the account.

[zar2]

Ok, do you have an ETA now?

[GatorMark]

Data is still being transferred to the new server. We anticipate having about 1/3 of the accounts online in about an hour and a half and the remaining accounts coming online as they copy over.

[zar2]

Could you please give an update? Email is more critical than web server.. do we have a warranty it will be restored ?

[ddbbtc]

Losing $$$, what is going on there?

[GatorJacob]

Team,

We have upgraded this server to a new beefier chassis (R710 - Dual-Raid, 12G Ram with 16 E5520 @ 2.27GHz Cores). The server is online and responding well. We have about 1/4 of the accounts on this server up and running from our backup systems, with the rest following as we get them transferred from our backup server.

The file systems should not be read-only, therefore emails should be flowing correctly. ( I have verified this, emails are flowing in the server). If your email / account is not working, most likely it's in queue to be transferred from our backup server. It should be online shortly.

We apologize for the inconveniences and the frustrations. We know how troubling this can be, but please know we are working on this as fast as the hardware will allow. We have accounts transferring from our backup servers, and being unpacked all at the same time. It will take time to get this done, but we are hoping with the new chassis that it will be quicker.

As always, we thank you for your patience during this time.

[ddbbtc]

Any way to give priority to a domain? I have a high priority client who is dying!

[GatorVincent]

At this time, we are currently transferring the data fro the /home partition. We are currently at the letter "P" as far as users go. I expect this to take at least 2 more hours for it to complete. Thank you all.

[GatorDaveC]

The final accounts are being extracted at this time. Every account should be on the server within the hour.

Please let us know if you are missing any data. We will keep the old server online for several days, in case data is needed.

[blanco]

Our main domain is completely empty...
Can't connect with FTP

[GatorVincent]

Blanco, can you please PM me your domain name or username? Thanks.

[kggerman]

Im missing some data too.
my primary domain is holdasociados.net, and the domain where some of my data are missing it's holdworkshop.com

thanks

[zar2]

What is the date of this backup? Just to notify customers.. what will happen with email that was sent today to these domains?

[GatorVincent]

Originally Posted by zar2

What is the date of this backup? Just to notify customers.. what will happen with email that was sent today to these domains?

This is from Late Sunday and Early monday (the backups). If you see that your account does not have data, please send a support ticket to support@hostgator.com so we may look into it. Again, thanks every one for your patience and cooperation.

[blanco]

So how come there is no backup for our sites?

[GatorBenF]

Originally Posted by blanco

So how come there is no backup for our sites?

I've replied to the ticket you had open in our ticket system. Please let us know if you have any other questions.