we shutdown Mustang due to drive errors....

[mark1234]

I also lost mail and cutomer on whm so how will i get my customers account back?

do i have to put them in manualy?

and when was the last backup on mustang server?

:-(

[123456qwerty]

Yes I also lost one WHM customer and all my customers account emails between today and last Tuesday.

Customers who updated their websites since last thursday have reported that their website updates have been reset to last Tuesday too.

[Hosting by Design]

I am in the same boat as everyone else ... I have missing content on all of the sites I updated (and clients have updated) since early in the week. Are we going to get that back?

Just need to know, cuz I don't want to put the work in to replace the data if it is going to be coming from a back up in the next few hours.

David

[GatorBrent]

The array degraded 3.5 days ago and appears to have stopped writing to array at that time. Anything in 3.5 days has been lost. There is no way to recover it since offsite backup is even older (about a week)

[davoice]

Ok... So I lost an entire website and an entire trouble ticket system and accompanying mySQL database due to this problem - including 3 days worth of trouble tickets (couple hundred) which are now a total loss.

Can you explain what is being done to keep this from happening again in the future? Seems like you have a major customer assurity issue to address since you're not doing nightly backups and you've just proven that you can't trust your RAID setup.

If the drive stopped writing to the RAID 3.5 days ago how was the server still functioning? And how was new stuff that was added during that time able to show up on the web and work just fine?

- Daniel

[bonzihostin]

"Everything is up and running with no data loss at all. The server is a little slow with playing catch up from mail etc. It should be completely back to normal tomorrow."

I guess not huh? I lost my work and that of my clients. And I am glad that at least you responded here because my support ticket question was never answered. I am sure your email inbox is full and I understand but right now it ME who looks like the idiot to my customers!

[VPC]

I have said it before, and I will say it again, RAID is not good enough HG.

I said this when I first started with you as a hosting co, almost a year maybe. I simply was waiting for this day as I knew it would come and I knew there would be data loss due to your policies of backup.

I am not complaining that the drive went bad, that happens, I am complaining about how you retain peoples data.

If you lose the raid....1 week old backups is NOT ACCEPTABLE! Time and time again, resellers have said they would pay for nightly backups, me included.

I see various servers over the months go down in the forums, and come back up with lots of activities in the forums when this is happening. Then, I start to read about data loss, backups from a week ago being restored, customers complaining, yadda, yadda, yadda. Yet, nothing ever happens....backups do not get re-strategized, nothing. It seems the norm and acceptable to HG that 1 week of data retention is good enough.

This is why I conclude that even as a reseller at HG, your websites better be small, non-important, static, etc... because if you have a site that has dynamic content and a DB that runs the site....expect to lose up to 1 week. Why? because HG says YOU (the customer) are responsible for the backups.

Funny, my mechanic does not ask me for oil when I pay for an oil change. My Dr. does not ask me for q-tips when I go for a check-up. The list goes on and on.

And do not bother responding with "most hosting companies...." - and instead, set the industry standard for excellence HG.

HG - pull your head out and re-strategize the backup solution you are offering, or lack there of.

[bonzihostin]

After emailing sales it appears you will credit for one month. And a personal thank you from me to you guys at Hostgator for doing that.

Can I just add I would not be banging on your door asking for a refund if the sites returned as they were because personally I wouldn't care too much afterall hard drives are not perfect right?

For me you have done the right thing and I still have some explaining to do to my clients but I am sure we will get over it.

It's also saved me possibly considering switching to another provider.

Thank you.

[VPC]

Sweet, check out MUSTANG today....

Server Load 239.76 (4 cpus)

Nice

[GatorBrent]

Mustang is having drive errors on drive 0 so we shut it down to prevent any problems. We are trying to get the drive swapped now as well as a complete chassi swap in case something else such as the raid controller is causing the drives to go bad.

[mark1234]

Can you please give us an estimeted time for this, my business is suffering now i have 2 customer want to leave me.

[amartins]

EDIT: Just saw your other thread regarding this server I guess it's explained now.

Hi Brent,

Originally Posted by Brent

The array degraded 3.5 days ago and appears to have stopped writing to array at that time. Anything in 3.5 days has been lost. There is no way to recover it since offsite backup is even older (about a week)

You probably have a ton to do but when you have a chance please explain. How can something like this be happening without any sort of system alerts to that fact (or are the guys at theplanet dropping the ball)?

RAID1 seems like a pretty decent (costwise) solution for what we pay but even so I don't quite get how this could have gone under the radar, was the problem maybe a faulty controler rather than anything else?

TIA.

Antonio.

[garymchu]

A 12 hour outage because it took you 10 hours to realise the wrong disc had been replaced and now this? I spent a full day restoring data to client sites that you said you hadn't lost and you do this to all Again?

I am assuming they ran tests after yesterdays Fsck up

This is waaaaaaaay from acceptable, we all kept relatively calm with you through yesterdays nightmare, i hope you dont expect the same again, I lose one client and I will looking for compensation for the loss

NOT IMPRESSED!

[davoice]

I agree... we've all been fairly calm. But now this is beyond inconvenient. I understand that problems occur. I'm an IT Director myself. But this is outside the limits of acceptability. It certianly underscores a considerable gaping hole in your disaster recovery planning.

- Daniel P.

[garymchu]

Can we have an ETA please, and not 2 hours if its going to be 12, we all have clients to pacify and keep informed.

[123456qwerty]

SO how long are you down this time?

[garymchu]

Just asked support to keep this board updated more regularly, and got a copied and pasted apology, is that the best HG can do to keep us informed?

I remind you HG you are not the only ones being yelled at by clients!

That is adding insult to injury.

[GatorBrent]

Been on the phone yelling since it happened. Here's what I have just been told.....


"(lcarter-05/01/2005 16:47:03):
My techs are going over with tested good drives and raid controller. I requested that they test the equipment before any changes are made.

That are aware of the time necessity on this. I made it clear that they need to be making updates in this ticket so that there are no questions as to the status.

I have just gotten off of the phone with my tech and they will be taking the server down shortly to get this resolved. If there are any issues they will be updating the ticket accordingly."

Last phone call to me said they tested the raid card and it was beyond a doubt the problem so it's being replaced.

According to the planet we should be up shortly.

Theplanet is worthless anytime it comes to anything hardware related, and there screwups are reflecting on us. We told them yesterday to swap the raid controller just in case however they did not.

We do not have physical access to the datacenter so when something hardware goes on we are at the mercy of theplanet. keep in mind we shut the server down this time on purpose after seeing the drive display errors we did this to avoid losing any data until the problem could be fixed which is being worked on.

[garymchu]

WHAT???????


"Theplanet is worthless anytime it comes to anything hardware related, and there screwups are reflecting on us. We told them yesterday to swap the raid controller just in case however they did not."

You are hosting our sites on a dta center that is no good with hardware issues??? you have to be kidding right????

[scavok]

that was a lot of info without an ETA....

how much data loss will there be this time? Like for real, not "no data loss as long as you don't update you sites or get any email but once a week"

if you can't give a real ETA can you atleast say yes or no as to having it up for business on monday morning?
????

[GatorBrent]

Anywhere you go you get people who know what theyre doing and people are clueless. They have someo talented people, but the ones we've gotten so far haven't been.

Theplanet is one of the most respected names out there and why we chose to go with them. But what has been happening with this server being fixed should not be happening. We are very upset with them! Just as you are getting upset with us.

[garymchu]

Ok lets do this again





Any chance of an ETA?

[GatorBrent]

I was told an hour. We shut the server down to avoid any data loss. All that needs to take place is the bad hardware to be replaced. There really should not be any problems, but we are at their mercy and anything is possible.

I'll be fighting for updates until we are back online, believe me.

[gbarrows]

Hostgator publicizes a guaranteed uptime of 99.9%.

This means a maximum of (365 times 24 times .001) 8.76 hours per year.

Interestingly enough thePlanet publicizes 100% on their opening page.

This would seem to me to indicate there is a plan on thePlanet's side to provide either hot spares or some other hot swappable means of insuring their clients (Hostgator in this instance) remains up.

I would like to know if, on our (Hostgator customers) behalf as well as their own, Hostgator management will be leveraging this outage to insist on a more reliable scenario from thePlanet. Make that a question????

[GatorBrent]

Network uptime and hardware are two different things. If we're down for a day theyll give us what a 10% refund? Which adds up to nothing compared to what we have to refund our clients.