trenton :: emergency fsck

[GatorNate]

Trenton is going down for a emergency fsck on the /home partition. While this work is ongoing http will be down, but the server will still do DNS, will still accept email - queuing it for delivery, and will still do remote mysql queries.

We will update this thread as it progresses.

[GatorNate]

trenton lost the /usr partition during the reboot. We have replaced the RAID card and cables and are rebuilding the /usr partition. We will get the box back online as fast as we can.

[GatorNate]

The /usr partition has been repaired and the server is back online. We are continuing with the fsck of /home now. A maintenance page has been posted and will remain up for the rest of the work.

[GatorNate]

we found some faulty hardware which has been replaced, we are continuing to repair /home and will update this thread as the work continues.

[GatorFord]

At this time the fsck of /home has completed but some users data was removed by the fsck due to the faulty hardware issue noted previously. We are currently reverting the machine's /home directory and user data to the data we have on the NAS server from this Sunday's backup (2/14/2010).

If you have an issue with data for your account on the trenton server after we've completed the restoration from our NAS server please submit a ticket to support at hostgator and we will do our best to retrieve any data we can for you.

I will post here as soon as I have further information about the restore progress. Thank you for your patience.

[GatorFord]

The restore of the /home directory has been a success. We were also able to retrieve backups for users which weren't on the NAS server from the resulting fsck files moved to lost+found. Please submit a ticket to support at hostgator if you have any remaining issues with trenton. Thank you.