Pantera :: Scheduled Drive Replacement

[GatorJacob]

Team,

pantera will be coming down tonight at 10pm CDT to have a drive replaced. This must be done to ensure the integrity of the data on the server. The estimated downtime should be between 10-15 minutes.

We apologize for the inconveniences and the frustrations that this may cause. Thank you for your patience during this time!

[GatorJustinM]

We apologize for the delay with the drive replacement. The server is still offline after swapping the drive and we are working with the datacenter to see what the issue is. Thank you for your continued patience!

[GatorJustinM]

We are currently running a fsck on /var which is about 50% complete. Hopefully we should have the server back online shortly.

[GatorJustinM]

We apologize for the extended time of the fsck but this is still running on livecd. Unfortunately, there is nothing we can do to speed up the process. We will update the forum when the server is back online.

[GatorNate]

The bad raid card caused file system corruption on many of the partitions and we are needing to repair all of them before we can get this server back online. We will get it online as soon as we can.

[GatorNate]

the fsck is continuing, until it finishes we really can't be sure exactly what the status of the server will be. We are continuing to work on this server and will update the ticket as soon as we get it back online.

[GatorNate]

pantera has finished its serious file system repairs and is booted back up. We are working to recover data that was lost and restore full service.

[GatorSLadewig]

I apologize, we currently have our best admins working to restore complete service for this server as quickly as possible. Unfortunately, we do not have a current ETA on how long at this time. I apologize about the inconvenience.

[GatorJosh]

At this time, we have most of the services online and responding as normal. We are still double checking everything on the server and we will post again when this is complete.

[oobjvb]

This is a disaster. I cannot get a straight, consistent answer from support. I have got to know when this is going to be back online - My clients are at a standstill! No website, no e-mail... And I can't even tell them when it is going to be fixed. This is completely unacceptable.

[GatorJosh]

We are sorry about the issues. There were major file system issues on the server and doing everything we can to get this fully resolved.

Currently the roundcube user table is corrupted and we have been forced to disable roundcube until this is corrected.

You may notice service restarts while we are working on correcting all the issues. Once they are all corrected we will update this again.

[flavoursmedia]

Guys! Its getting on my clients nerves. They are alreading hitting me with so much mails asking whats happening. Its already more than 10hours. I don't think this is really acceptable. Is there no backup plan for emergencies like this??? ecommerce based sites are down for hours now. please do something fast.

[sammie]

I'm getting bombarded by clients as well. Business can't take another hit by loosing clients who are pissed that their sites and email is GONE. It's almost 12 now can we get another update at least?

[oobjvb]

I too cannot believe there is no backup plan for such an emergency. I simply cannot accept that there is no timetable and that this was not avoidable. For every client hostgator is dealing with, their client is dealing with dozens of their own clients. And we cannot give them an answer. Sites and email have been down for more than 12 hours folks. There is no excuse for there not being an adequate backup that can be restored when something like this happens. I and many other of hostgators clients are getting CRUCIFIED out here - and we can't even give any answers. Untold numbers of businesses, productivity, profits and e-mails are being lost. We rely on sound hosting practices, but our clients rely on US. WE NEED ANSWERS NOW!!!

[devries]

In the +4 years I've been a hostgator customer I've never experienced anything on this scale. And typically the most important of my clients has still got their email down whilst the rest seem to be working.
I'm loosing my clients! I'm in GMT which means everything has been down all day - this is very serious for me.

[oobjvb]

Are we going to be compensated for lost business and clients? Why will no one give us a time frame? I do not get the idea that Hostgator comprehends the severity of this situation. And some of the support personnel I have spoken to today have been downright flippant or, worse, rude about it. These are our businesses - our livelihoods... Is an estimated time frame too much to ask?

[GatorJosh]

We are doing everything possible to get this back and running as it should. The drive had to be replaced last night and the array is rebuilding causing high IO on the drives. The problems we are seeing at this time is that some databases such as roundcube is corrupt. Whenever those dbs are being accessed it is crashing mysql. The loads are also spiking because of the raid rebuild.

We do not have an exact ETA. This has obviously taken longer than expected. We will address all questions but need to focus on getting the issues resolved first. We have locked this thread for now and will continue to post updates.

[GatorJosh]

Databases are currently being repaired at this point. We are going to attempt to bring up apache to allow pages to be served.

[GatorJosh]

Because of the crashes, we have been forced to disable innodb on the server for now. Most of the sites should be loading at this time. There will be slow page loads while the raid is rebuilding.

We are also still working on repairing databases. You may get a db error on the page until this is fixed.

[GatorJosh]

In order to repair dbs, we have been forced to put up a maintenance page for all sites. You should be able to send/receive emails using an email client. We have also been forced to stop cpanel to prevent server issues.

[GatorJosh]

At this point most of the services are running, including apache, mysql, and email. We are going to monitor the server very closely and will enable more services such as cpanel soon as we feel that the server is able to handle it. You may see some higher server loads or longer page loads until the raid rebuild is complete.


This problem was caused by by a failing drive last night. The failed drive caused file system issues, including database issues and missing libraries. We were forced fsck each partition, then sync the libraries from an image server and repair the dbs. This took an extremely long amount of time because the raid was rebuilding from the drive swap. During the 3ware drive rebuilds, the servers can run very slow causing the issues we have seen.

We are very sorry about all the issues you have seen here. We are working on a more efficient process to better handle this in the future which includes some system and kernel changes.

[GatorVincent]

At this time, we are adding more ram to the chassis to help make the raid rebuild more efficient. We sincerely apologize for this but this will help other services to run more efficiently while the raid rebuilds in the background. We are working as fast as we can to bring the server back up into a productive state. Thank you.

[GatorVincent]

At this time we are still working on getting Pantera up, we apologize for the delays but we are working non stop to get the box in a productive working status.

[GatorVincent]

At this time, the box is back up and serving requests, there may be some isolated events where you may receive timeouts. We apologize for this, the box at this time is under normal load and is now efficiently rebuilding the array.

[GatorSLadewig]

Hello,

At this time the rebuild has completed, and there load times should be back to normal for sites. Please let us know if you experience any issues, or have any questions.