we shutdown Mustang due to drive errors....

[faze1]

I'm on the mustang server and I'm still not back up from this episode. I was out of town yesterday and luckily I did not hear anything from any of my customers until this morning.

All my sites are still down and I can not access email. I offer my customers 99.9% uptime and at this point I have lost all my revenue for this month and it hasnt even started yet. How can my sites be down while everyone else seems to be back up?

I left a message on voice mail and no one has been online for support. I'm really hoping I'm back online soon so I can asses the data lost.

Thanks,

[garymchu]

Guess what, you guys offered us a month which is nothing compared to what we have to refund our clients we know that one well.

Then of course there is any clients that are lost, then we start multipling permanent lost monthly income, because "The planet are wortheless when it comes to harware"

You were told an hour, just like yesterday its well over that now, my support system is going mad and I have nothing to tell my clients. Once is ok but 2 days on the trot sheesh.

Inclidentally I talked to the planet just know on the phone and they tell me that a drive replacement shouldn't take long to rectify.

I'm dumbfounded. I will certainly never pay for hosting for a year again

[scavok]

hostgator needs to go shopping

I'm glad I backed up everything last night after I reuploaded all my newwest clients. Will I have to do that again?

[GatorBrent]

garymchu were in the same boat as you........ everything you said applies to us. You have just as much power as us to get this fixed. I've been on the phone all weekend with them calling all my contacts.


Why wasn't it completely swapped when I requested yesterday? I dont know
Why did they swap the wrong drive yesterday when we clearly told them which one? I don't know

Why is this taking hours when swapping should take 30 min? I dont know

what it looks like........ They have staffing problems and everyone there is doing the best job that they can but overworked and underpaid.


what we are going to have to do......... I'm going to have to fly to texas and hire someone to handle our hardware problems correctly when they come up.

[faze1]

Believe me, I understand that math. I am in a similar situation, the credit for this month does not do much to offset the money I need to reimburse my clients.

I'm praying there is minimal data loss, as almost every client site is still in the process of development and are being worked on constantly. Explaining downtime is one thing, explaining a weeks worthe of workj down the drain is another.

Please give us a realitic eta soon, so we have something to tell our clients.

Thanks,

[GatorBrent]

It is back online now. I'm waiting for news to make sure this will not happen again and to confirm the real problem has been fixed.

[GatorBrent]

they took it back off line not sure why yet.

[123456qwerty]

Pinging 67.19.170.34 with 32 bytes of data:

Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 67.19.170.34:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

[scavok]

update? (sorry just a lil anxious here)

[GatorBrent]

up again. I was told raid was replaced so it looks like they are done with whatever they did. I'm trying to find out what exactly was done so we can be sure it is fixed this time.

[GatorBrent]

"(npranivong-05/01/2005 19:18:00):
I have swapped out the 3ware RAID controller card and the 200GB Drive on port 0. 3Ware's did not report the drive to be non-usable, which is usually the case when a drive has is bad. However, when I booted your server prior to the hardware change, your server was reporting

scsi1: AEN: Warning: Sector Repair Occurred: Port 0. This could be due to ECC errors on the HDD."


here's what looks like happened.........

They replaced the bad drive with a brand-new drive that also was bad. how and why we are investigating with them.
we should not see any more issues related to this going forward.

[VPC]

Apache no workie...sigh

[GatorBrent]

We were able to figure out exactly what happened!

The server was up and running fine up until four days ago when one of the drives in the raid failed. We had it scheduled to be replaced over the weekend when traffic was at its lowest. By coincidence the working drive also completely failed the same day it was scheduled to have the other drive replaced.

When they took it off line they replaced the one drive and that's when we were waiting the entire day for the fsck. That drive was completely bad and could not be salvaged. The technician who was working on it believed they replaced the bad drive with the same drive. But in fact they did not screw up! both drives were bad!! the one drive was completely unusable the other was in the process of failing. That is the one that went down today and was replaced. It had enough life left in it to continue on and give us a full backup on another drive. However was the one erroring today which we replaced.


Nobody screwed up, just no one caught on either that both drives had failed.

This is extremely confusing so I'm not sure if anyone here will understand what I wrote but here's a quick break down....

two drives were operational 0, and 1
drive 0 failed and the server was running on drive 1
drive 1 failed leaving the server with 0,1 failed.

they replaced one Bad Drive during the first outage, and then another in the next when we shut down the server.

We have only had a couple drive failures out of the hundreds of servers we have to date. I have no idea what the odds of 2 drives on the same server failing within days of one another!!! crazy!

[GatorBrent]

We were able to figure out exactly what happened!

The server was up and running fine up until four days ago when one of the drives in the raid failed. We had it scheduled to be replaced over the weekend when traffic was at its lowest. By coincidence the working drive also completely failed the same day it was scheduled to have the other drive replaced.

When they took it off line they replaced the one drive and that's when we were waiting the entire day for the fsck. That drive was completely bad and could not be salvaged. The technician who was working on it believed they replaced the bad drive with the same drive. But in fact they did not screw up! both drives were bad!! the one drive was completely unusable the other was in the process of failing. That is the one that went down today and was replaced. It had enough life left in it to continue on and give us a full backup on another drive. However was the one erroring today which we replaced.


Nobody screwed up, just no one caught on either that both drives had failed.

This is extremely confusing so I'm not sure if anyone here will understand what I wrote but here's a quick break down....

two drives were operational 0, and 1
drive 0 failed and the server was running on drive 1
drive 1 failed leaving the server with 0,1 failed.

they replaced one Bad Drive during the first outage, and then another in the next when we shut down the server.

We have only had a couple drive failures out of the hundreds of servers we have to date. I have no idea what the odds of 2 drives on the same server failing within days of one another!!! crazy!

I have closed the other two threads if you have any questions please post in this one.

[VPC]

Yeah, no MySQL DB's...can't wait to hear this one.

Anyways, I am in IT and I have never told a client that we will let their server run on 1 drive until the weekend. I cannot imagine a client that would say "No, do not replace the drive asap and leave us vulnerable until we are not so busy."

Nice way to gamble with someone elses livelyhood. Simply the dumbest decision I have seen in IT.

[GatorBrent]

it's fine. some services are being overloaded from it being down so long and everyone slamming the box give it a little time to catch up.

[GatorBrent]

VPC running on one drive is fine. 99% of the hosts out there run on one drive 24/7/365


Very few companies have a raid set up like we do. The chances of two drives failing within a few days of one another is extremely rare. We also had full backups on our disksync system that was taken 7 days prior.

[VPC]

"VPC running on one drive is fine. 99% of the hosts out there run on one drive 24/7/365"

Exactly why someone that choses a company who advertises RAID like HG then that company says 1 drive is ok because others do it, is why people should not consider HG the standard of excellence, thanks for pointing that out Brent.

Saying that 1 drive is ok, just kind of proved you wrong with this weekends events, thanks again for pointing that out. Guess you will not consider changing out a drive next time and roll the dice with clients data again. Nice Brent.

To address your backups, 7 day backups is nothing to promote. 7 days is a long time for dynamic MYSQL websites to go back to. They do make backup solutions that can handle the amount of data on the HG servers. If you want to boast about something, get nightly backups and make HG a leader in hosting, or simply keep saying "other hosts do the same"

[davoice]

So what we customers who got burned on this would like to see is:

1) A free or paid nightly backup service. So we aren't at the mercy of a 7 day interval.

2) An *easy* way for us to backup all the stuff we have - all the email configs, mySQL databases, etc. - on an ad hoc basis when we feel the need. (Cpanel's backup util isn't quite there.)

3) Faster remediation and notification to customers of failed drives - and failures in general - in the future. I agree with the previous posters... as an IT Director myself, I would never let a failed drive go more than 24 hours. That is simply asking for trouble. I would also take a backup snapshot of the server as soon as you find out a drive has failed. That way you're having your arseparts flapping in the wind if the 2nd drive fails before a replacement can be brought online and synced. Taking a backup when the first drive failed would have saved your butt (and all of ours) here. But we didn't have the luxury of taking a backup ourselves b/c we weren't warned a drive had failed.

4) Make a suitable apology - both written and financially - to all your customers affected by this issue. We've basically lost an entire 7 days of work due to this problem. I personally lost $450 of consulting time we had paid for getting our business' web trouble ticketing system up and running from Wed-Fri before the complete failure. For many customers that is equivilent to shutting their doors. While hardware failures aren't your fault, you are responsible for them. That's part of the liability of running a hosting business. Even though you can blame ThePlanet for assisting in you having a thick layer of egg on your face, it's still solely your job to clean it off.

- Daniel P.

[garymchu]

I agree totally, am researching an alternative service.

Several things throughout this incident ruined my opinions of HG.

Technical issues aside:

1. "The planet are not good with hardware issues" (or something to that effect) Nice to know.

2. Call this petty if you wish, but what really blew me away was my request for this board to have more regular updates from admin being met with a copied and pasted apology, from live chat support. That was a complete and utter insult!

My support team have spent all this down time communicating one on one with our clients nightmare as that was and not with copied and pasted replies.

3. The changing stories and the allocating of blame also does not make for good support.

Our clients expect fixes form us when there are issues.

We expect fixes from HG

HG expect fixes from the planet.

Seems the top of that tree is broken, I certainly am not prepaired to offer a service to my clients without adequate backup.

[VPC]

I would pay for a nightly backup. Hey Brent, do your servers in your internal network at HG run on nightly backups? Just wondering because I have never had a client that runs anything less than nightly backups. Why do hosting companies think anything less is "acceptable"?

Why does HG continue to use the "well the other guys do not have it" or "well the other guys have the same thing" damn, what happened to striving to be the best?

Let me tell you something, I would pay for items 1-4 in Daniel P. post because my clients pay me to do 1-4 for them. I would not want anything less. Is this a money issue or just an acceptable issue for HG?

[GatorBrent]

It takes about 48 hours to complete a full backup. You can not do something daily if it takes longer than that to do

If anybody wants daily backups they're welcome to purchase a dedicated server, and assuming they do not have much in disk space used it will perform a daily backup.

to restore a full backup.... takes at least a day. had we not recovered the raid the restore alone would have taken much much longer.

An array rebuild takes close to 24 hours writing disk to disk.


backups are the heaviest thing you can do to a server. if I was to start a full backup on the whole server now load would go to 20+ If we did it daily load on the server would never go below 20.

[GatorBrent]

Do a virus scan on your computer and watch how slow it will go. Backups on a server has an equal effect.


Daily backups is not sufficient either. Say something happened while the site owner was sleeping and the daily backup overwrote itself.

[VPC]

"It takes about 48 hours to complete a full backup. You can not do something daily if it takes longer than that to do"

I do not know what you are using for backup, but that is horrible. You do know that TANDY's are no longer a viable solution don't you?

Thanks for letting me know about the effect on my PC with VS. I am quite capable being in IT of knowing what loads by apps are. It is not a problem if the load is throttled and the right backup hardware is used to cut your 48 hr time to a 10th of that. I do it everyday.

[GatorBrent]

VPC provide a viable solution and we will consider it.