acura hard drive needs to be replaced

[GatorBrent]

3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.


We need to take down the server for a few minutes to swap out one of the servers drives and have the array rebuilt. We have already spoken with the datacenter and are planning to do this tonight.

This needs to be done before the drive completely crashes and causes damage. It also is causing the server to be unstable at the moment. The datacenter assured us they can rebuild the array while the box is online and the good drive will not be at risk. We have done this on a few occasions without any problems. If you have any questions, please let us know.

[Ming Keong]

Thanks for the update. I hope this resolves the instability problem as I'm seeing multiple downtimes each day.

[salubrium]

These people who's sites have been suspended.. I don't think their sites are causing problems so much to say that something in the configuration of the server has changed (possibly the 4.3.10 php upgrade?), or something else that is causing the server load to ramp up very high.

Have you check the second hard drive / raid controller? It would be wise to do so, especially if Acura was purchased around the same time as Supra from the same manufacturer..

Interesting that your post about the hard drive came 1 hour after my post above about Acura. I was a little afraid of sounding like a smart-ass but after seeing the hell that you have gone through with Supra, I thought it might be a good reminder to do the checks.

So I beg the question: Did you check it after my post? Either way, I am glad that you guys have picked it up before there was a major issue and I do hope it solves the load issue that Acura has been having.

I was driving today after reading the posts about Supra.. and for me, Brent sounds quite genuine and was pretty good (and honest) in the feedback he was giving the customers. I know this is something I appreciate as a customer. Just keep me informed, so I don't look foolish with customers.

But then I thought.. 1000 sites on one server.. (I don't know if this is an accurate number).. What other type of business would have so many people affected because of one $100-400 part?

I'm glad I wasn't in your position Brent.. I can imagine you waking up every hour.. I'd be doing the same.

And for those that have been complaining, go signup with Sultanhost for a while.. I had my site down for 3 days, no response to my emails, no response on their forums.. nothing and they don't have live support. Now that makes a man nervous.

Anyway, GH might not be perfect but I think they do a decent job as a hosting provider.

[kory]

Well, this might explain the webmail issue.

Seriously, I noticed Acura has been performing very slowly all last week and this week. I thought it might be a network issue but if it's a hard drive issue, I'm a bit nervous, especially after reading the Supra fiasco.

My sites were "suspended" too but Brent brought them right back up as he didn't know why they were suspended (thx, Brent). I'm beginning to think, too, that perhaps it wasn't client sites causing the issues.

Just glad to know that it's being taken care of.

Thx much, guys.

Regards,
Kory

[airkat]

Thart just sucks for the sites that had to pack up and go to another host because HG thought their site was loading down the server...

[Ming Keong]

Originally Posted by airkat

Thart just sucks for the sites that had to pack up and go to another host because HG thought their site was loading down the server...

I wish that they will reconsider the decision..

Anyway, the load is pretty bad now. I just hope that the issue will be resolved soon.

[kory]

Anyone else having up/down problems w/ Acura? My sites are all down now, including my clients.

I'll admit that I'm in the process of moving to a new provider. I didn't want to but, at this point, I don't have a choice. The migration's almost done; just have to switch the DNS servers over.

FWIW, I wish HostGator the best of luck and hope that these issues are resolved soon for their and their clients' sake.

Regards,
Kory

[GatorBrent]

The datacenter jumped the gun and has begun the swap now without final confirmation from us. It is well needed so maybe it's good to get it done with however we would have liked to waited to a less peak traffic time.

[GatorBrent]

I have called again to confirm that there is no risk to the good drive and was assured there was not. Due to the supra issue we are very nervous regarding this issue =)


We should be back online in 20 minutes or so if not I will call to find out what is going on.

[GatorBrent]

"The second drive has been replaced, and the array is currently being rebuilt. It will probably take some time to complete, but your server is up and functional. I will be monitoring the progress throughout the day. "


>>>>>>>> Big sigh of relief

That only lasted a few minutes! The server array will probably take 24 hours to rebuild. During this time load will be higher than it should be and it possibly Might cause services to be less stable than they should be although more stable than they were.

It's possible there's something else also causing issues on Acura but now with the hard drive fixed we should have no problem tracing it and fixing.

[kory]

Actually, no relief from me. My accounts have been "suspended" again and I can't access cPanel.

I sent a trouble ticket in again. Did I do something wrong again to warrant the suspension?

Regards,
Kory

[GatorBrent]

Server load skyrocketed to 70+ we found three sites to be the cause and they are now permanently banned they are the problem and must be kicked to solve it.

It was hard to track down due to server hard drive issue. It's hard to blame sites saying "oh hey by the way it could be the hard drive"


Things on Acura have been resolved.

[kory]

Originally Posted by GatorBrent

Server load skyrocketed to 70+ we found three sites to be the cause and they are now permanently banned they are the problem and must be kicked to solve it.

I have three Mambo sites on that server and two of them have been "suspended." Inquiries to both support and billing go unanswered. The support ICQ told me to contact billing. If my Mambo sites are the caused, I'd like to know about it. I'm just trying to get access to my cPanel and can't.

Regards,
Kory

[ztechpc]

according to your other post in the supra thread you removed the monitoring software from you servers. whtas this about then and why did supra not have this warning?

Originally Posted by GatorBrent

3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.
3w-xxxx: scsi0: AEN: WARNING: Sector repair occurred: Port #1.


We need to take down the server for a few minutes to swap out one of the servers drives and have the array rebuilt. We have already spoken with the datacenter and are planning to do this tonight.

This needs to be done before the drive completely crashes and causes damage. It also is causing the server to be unstable at the moment. The datacenter assured us they can rebuild the array while the box is online and the good drive will not be at risk. We have done this on a few occasions without any problems. If you have any questions, please let us know.

[GatorBrent]

Supra was not the same issue and when we were in it a day prior checking things out there were no errors.

Supra was a freak incident being a raid controller / hard drive failure

[salubrium]

I have three Mambo sites on that server and two of them have been "suspended."

I also run a number of Mambo sites. I am now running 4.5.1a on my sites and I am using caching as well. I'd like to know what it takes for a Mambo site to become a resource hog.

Kory, were you using any particular modules or components that could have attributed to this? How many simultaneous users do you normally have? I have a Mambo based site offering a file base of 870mb. I force users to register to download the files. I'd be pretty annoyed if it was suspended considering the hours it takes to upload all those files over 128mbit line.

Brent, when you say three sites are causing unacceptable load. What is it exactly causing that load? Is it mysql, apache, php, perl, memory, disk access, bandwidth?

We don't really have the access to 'troubleshoot' our resource usage but if we at least have some idea, then it can help us rectify the problem. Save you losing a customer (and having to find a new one) and we feel we are working with someone who 'cares' about our business.

What if you DC just pulled the plug on one of your servers (permanently) because you were told you were consuming too much bandwidth and you can't even access the server temporarily to move the 1000 sites hosted on that machine? - me imagines a tight spot..

From what I have read on the forums, when people's accounts get suspended, the main thing they want is a notification that it has been suspended. I agree that if a site is causing a server load of 70.. then hey, you have to stop it straight away for the sake of other customers but you should definitely EMAIL them at the same time you suspend their account.

It would be nice if cpanel included an automatic notification that your account was hogging resources.. Is there any GPL server side software that monitors an individual users resource usage and can email them if it causes such server load with the details of the offending process / pid?

At least then users would get a warning of some type and they can take action to correct it prior to being suspended.

[kory]

Originally Posted by salubrium

I also run a number of Mambo sites. I am now running 4.5.1a on my sites and I am using caching as well. I'd like to know what it takes for a Mambo site to become a resource hog.

All my sites are running 4.5.1.a and utilize caching as well as I'm sensitive to overusing the server. The last thing I wanted to do is cause problems for others.

But, I'd be interested in knowing too for Mambo to become a resource hog, especially since they were installed via Fanstastico. Perhaps it should be removed from Fantastico if it's such an issue.

Originally Posted by salubrium

Kory, were you using any particular modules or components that could have attributed to this? How many simultaneous users do you normally have? I have a Mambo based site offering a file base of 870mb. I force users to register to download the files. I'd be pretty annoyed if it was suspended considering the hours it takes to upload all those files over 128mbit line.

Seriously, none that I could think of, salubrium. My sites average about 30 simultaneous users each.

Now, I will say this. I recently installed another app, WSN Links, that might have had something to do with it. I'm not planning on using it though as I think that might have been the problem.

Originally Posted by salubrium

Brent, from what I have read on the forums, when people's accounts get suspended, the main thing they want is a notification that it has been suspended. I agree that if a site is causing a server load of 70.. then hey, you have to stop it straight away for the sake of other customers but you should definitely EMAIL them at the same time you suspend their account.

That's all I'm suggesting too. I had to send an email to HG to be notified that my sites were causing the problem (yes, it's confirmed). I'm a bit concerned about how my four to five sites could cause a problem on a quad processor box with almost 1,030 other sites. But, I understand to be shut down if I'm the issue.

Now, I just want a few minutes of cPanel access to grab my one database so I can finish the move to my new provider.

Regards,
Kory

[kory]

Well, I'm really trying to be diplomatic as I've sent a message to support three times over the last two hours and no response.

All I want is access to one of my clients' cPanel so that I may grab the backup and move on. Is that too much to ask?

I hate to resort to the forum with this but I'm not sure what else to do. The last thing I want to do is create a fuss but someone suggested to me to have my attorney contact HG. I'm not the kind to cry to lawyers but, man, I just want to take my data and move on.

Regards,
Kory

[kory]

In all fairness to HG, they finally emailed me back with the FTP locations of my site backup data in tarball format. Thanks for doing that. I greatly appreciate it.

I'd like to wish HG and all of its clients good luck and a prosperous New Year (no, I'm not being sarcastic; I really mean it! That's how I am when I'm not irritated. )

Regards,
Kory

[GatorBrent]

You are a good sport Kory! Sorry for the delay on getting it to you, things are still crazy.

It's too hard to say which of the four sites caused the most problems but they were the heaviest users on the server from what we were seeing. We haven't had any more problems on Acura and the load has been below 2 all-day. We don't see any problems with mambo but on extremely active sites such as 30 users online at a time things can be busy with anything you are using .

I t is also possible that one out of four sites that we suspended caused more damage than the other three combined. It is also possible all four sites played an equal part. We have a duty to server as a whole and did not have the luxury of experimenting. How much damage each site was causing. All sites were heavy, and all should be in a vps or dedicated solution.

[gtoddv]

I am glad to see the situation with kory resolved amicably. But the question still persists, why doesn't hostgator notify folks about suspended accounts? Or better yet, isn't there a way to throttle server resource usage?

HG appears to have a great business model. I am very impressed with the quality of service. I think you guys have an opportunity here to really make a good impression in the hosting community (I mean more than you already have). Not all situations can come to closure as well as kory's did, but with a few tweaks here and there, you could certainly increase the odds that they do.

For those of you out there thinking of running a community portal, Drupal has a throttle feature that could help you stay under the radar. Until there is a way to know how your site is affecting the server, this is a solution that might keep your account intact.

[kory]

Originally Posted by GatorBrent

You are a good sport Kory! Sorry for the delay on getting it to you, things are still crazy.

No problem, Brent. Thx much for the explanation. I greatly appreciate it as I'm sure others do too.

I also posted my neutral experience in the other forum:

http://forums.hostgator.com/showthread.php?t=2086

Good luck to you, the entire HG staff and its clients!

Regards,
Kory

[GatorBrent]

We always notifysuspended sites, every now and then we forget to send an e-mail out, or the e-mail is sent to the same e-mail that was suspended, or spam filters block it. We suspend up to 10 sites a day from spamming, crashing servers, or trying to hack the server.

With normal hosting this is a easy task when it comes to reseller hosting it is much harder, because you have more sites and when we contact the reseller their partner or web host does not contact them.