Pulsar :: Drive Replacement :: 3-22-2010 :: 22:00 CDT

[GatorLBrower]

pulsar.websitewelcome.com will be coming down tonight at 22:00 CDT 03/22/10 to have a drive replaced. This must be done to ensure the integrity of the data on the server. The estimated downtime should be between 10-15 minutes.

We apologize for the inconveniences and the frustrations that this may cause. Thank you for your patience during this time!

[GatorLBrower]

This has now been completed.

[GatorJosh]

The drive has been replaced but the array is currently rebuilding. You may see some slower speeds during this time. It is currently at 57% complete and we will be monitoring the server during the rebuild to keep them stable.

[GatorFord]

The rebuild is at 96% at this time and is close to completion. We will update this as this continues.

[NBClark]

Has this finished yet? This is getting insane. Over 36 hours for a RAID rebuild? This server has been plagued with "overload" issues as of late - causing significant and serious e-mail delays. (See Ticket #EHH-13033632)

I've got very unhappy clients because of the issues over the past month or so, and now this. I realize you have to do what you have to do in order to keep the server up and running, but my domains have been SEVERELY impacted for the last day and a half - including not being able to log onto the POP server from 11p on 3/22 until after 2p on 3/23. Now it appears to be happening again today. I've been monitoring the server all day and it seems to be hovering at 35-55 CPU load and 72% memory load. Now it's spiking between 130-265 CPU! My phone's ringing off the hook with complaints of being unable to log into POP, messages that appear to be sent but the recipients aren't receiving them (I'm guessing delay again due to overload), etc.

Can we get a firm timetable on when ALL of these issues will be resolved? I'm ready to start looking for another host because of the lack of consistency on this server.

[GatorFord]

NBClark,

I sincerely apologize about the issues this is causing for you. The 3ware card's unfortunately take a bit longer than the PERC's for rebuilds, and the time required for this rebuild was exacerbated by the fact that kernel upgrades were performed last night for which this machine was included.

During the subsequent reboot the server had to restart the actual rebuild which is why this is taking longer than normal. I do sincerely apologize, the rebuild is at 97% at this time and should complete hopefully within an hour or two at most at this rate.

We are working on a new system to ensure machines which are in a rebuild state are not rebooted during anything like kernel upgrades. Please let me know if there's anything else I can help you with or answer.

[Committee_Chair]

I'm sorry, but I have to add my voice here.

My clients have been calling me since Sunday, websites up and down, email not working or working but not forwarding. When my email does work, I've received over 600 emails in the last 48 hours generated by the web applications not connecting the MySQL database.

I don't really resell to the general public, I recommend my own hosting to my web design and maintenance clients that I talk to on a weekly basis. I'm horribly embarrassed. Had I realized that there was no redundancy built into your process of fixing and rebuilding drives, and that you'd be rebuilding a drive on my live data, I doubt that I'd be hosting with Hostgator.

This is unacceptable at any price. 99% uptime means that we can be down 3.65 days. I'm at the end of the 4th calendar day with no light at the end of the tunnel.

I can't have this happening now, and I have to make sure it doesn't happen again. I really don't want to go through the trouble of moving to another host, but if it means I'll avoid the embarassment, and my clients won't feel like they're loosing business, I'll do the work.

What can hostgator do for us, besides continue to use the same "process" of fixing hardware on top of our live data?

[pmeyers83]

Have all of the issues been resolved?