Scheduled maintenance and email/site unavailability

[kanjigirl]

HG replaced a drive on the Lebaron server at around 9:00pm server time Wednesday. That was reported under the 'scheduled maintenance' forum, and that's great that we were notified.

However, I had no idea that this meant that sites and email for accounts on Lebaron would only be intermittently available for nearly 2 days. No clue. My ~45 clients on Lebaron were not happy with me to say the least.

So if this was scheduled in advance, and it was known that it would have this effect, why not mention that? Or even better, take care of it over the weekend rather then during the middle of the work week?

An emergency, sure, no problem with that. But since it was planned, I was really unhappy about having to explain to my clients what was happening and why it was proceeding so slowly.

Please give us info like this up front if you know it's going to happen - if I had warned my clients that they would have site downtime and nonfunctional email for ~48 hours, that would have helped somewhat, although it's still not clear to me why something like this was scheduled in advance during the week.

[GatorDHanna]

Hi there,

I apologize for the inconveniences.

The actual drive replacement usually doesn't take much longer than we say it will (10-30 minutes). When they are replacing the drives, the server is physically offline and therefore down.

Once the server comes back up, the RAID array has to rebuild, which takes 24 hours or so. During this time, the server will probably respond slower than usual, which is probably what you noticed and why it appeared to go up and down intermittently. This slow down doesn't occur with all of our servers, but it does affect a few of them.

I'll talk to a few people and see if we can start mentioning these types of things in addition to the pure server offline ETA.

Does this explanation help?

[kanjigirl]

That does help, but it was more like 40-44 hours than 24.

One other issue - since Wednesday I noticed a huge increase in spam on two of my accounts with you. I went in and discovered that SpamAssassin has been disabled. Now I'm getting calls from some clients letting me know about a big increase in spam and every one I check has a disabled SpamAssassin. More work for me, but unfortunately it's not billable.

[GatorDHanna]

I'm glad the explanation helped and I want to apologize for the inconveniences once again. I know the intermittent slowness and downtime can be frustrating.

I checked into the issue with Spam Assassin being disabled with one of our senior administrators. While the RAID array was being rebuilt, Spam Assasin was temporarily disabled to lower the disk I/O (and get the array rebuilt sooner) but didn't get re-enabled for some reason.

If you run into any other issues, send us an email and let me know what the ticket ID is. I'll ensure that one of our best administrators checks into it for you. This is obviously not something that happens very often and I apologize about the trouble.