gator222 :: Outage

***GatorDaveC*** · #1 10-06-2007, 04:40 AM

One half of the RAID Array on this server has failed. One drive became degraded tonight, which isn't unusual. The other drive that it was being mirrored to started to get uncorrectable ECC errors which is pretty rare to just happen all of a sudden.

I don't have an ETA right now, I'll update you once I have more news.

***GatorDaveC*** · #2 10-06-2007, 05:57 AM

The problem is under control for now, I am moving the files to another server at this time. All of your data should be there, but I have noticed one area of the OS that was damaged.

The drive that was giving ECC errors shows that it has unrecoverable sectors now.

All services have been online for a while now. We should expect to have this servers drives swapped this morning or early after noon.

Tomer · #3 10-06-2007, 07:45 PM

Would of been nice to get an email about this with out having to find out that my emails haven't been arriving. Still nice that we see you guys are working on it.

First server issue in over a year, I guess I can't complain huh?

Is any data loss expected?

Thanks! Keep up the good work.
- Tomer

***GatorDaveC*** · #4 10-07-2007, 12:52 AM

Issues like this rarely happen. Arrays go degraded from time to time, which just needs a drive replaced. We constantly monitor any S.M.A.R.T errors on the drives.

This issue was a little unusual since one drive was degraded for less then 24 hours, but one of the other drives in the RAID10 had unrecoverable ECC errors.

You actually should be receiving e-mails without any issues, since the file systems are Read/Write at the moment. The data is still being migrated to the other server at this time.

Quote:

Originally Posted by Tomer

Would of been nice to get an email about this with out having to find out that my emails haven't been arriving. Still nice that we see you guys are working on it.

First server issue in over a year, I guess I can't complain huh?

Is any data loss expected?

Thanks! Keep up the good work.
- Tomer

***GatorFord*** · #5 10-07-2007, 11:52 AM

The data synchronization has been completed and at this time. We're now working with our datacenter to physically migrate the hardware over. As the machines are at two different physical datacenter locations this will probably take around an hour or so, though gator222 won't actually go offline until the hardware is all at the location and ready to be swapped.

Thanks for your patience and we'll keep you posted.

***GatorFord*** · #6 10-07-2007, 01:35 PM

gator222 has been swapped out and is back online. Let us know if you see any problems. Thanks!