audi.websitewelcome.com update

[GatorBrent]

It appears the raid1 had some type of failure today instantly corrupting the drives with bad data. We are having to restore from our disksync backup which last had a successful backup on may 9th.

We are currently doing this now and believe it will take 5-6 hours for the restore to complete.

[GatorBrent]

we will keep everyone posted......

[Lumis]

You gotta be kidding me? 10 days worth of information gone? How can a RAID corrupt both drives....is this the best case scenario? We lose over 12 hours and we lose 10 days worth of data, let alone what happens to emails, which every client I have is very unhappy about.
It was bad enough reading about other threads where this has happened, like the Mustang server, so how can it happen again??
Most unhappy reseller.....

[shurik]

thats exact thing that happened to me with my previous hosting provider and that is why I switched to something that supposed to be more reliable. Now it doesn't look like you guys have anything to back up your claim guaranteeing 99.9% uptime.

I wish my customers only simple problems like email outages or some lost data.... Every hour the site is down my customers are loosing thousands of dollars.

[GatorBrent]

Still restoring.....

Hardware issues happen, we def aren't having a fun time during this.

[bshell]

Thanks for working so hard to get this back up.

[kwe]

This is totally inexcusable. I heard the same story from you guys last week I requested a restore from backup. It took 3 days and the backup was 2 weeks old. Why don't you have an images of your server so you can quickly and easily retore a server and then simply restore the domains from a CURRENT backup? This is standard procedure for any network.

[Lumis]

There has now been 15hrs downtime this is unbelievable if I had read that hardware failures may take up to 15hrs to fix I would never have purchased hosting here, but not only that to go back 10 days on the backup restore is even worse total cost of this downtime may put us greatly out of pocket both short and long term.

HOW LONG IS IT GOING TO BE ?

(PS Thankyou for trying hard but we need it fixed not these excuses)

[Freakindj]

WTF

We were seeing a gradual restoration of the server functions.

http://www.hostgator.com/s/

Now we see a total failure

Any chance of you guys making some progress and not coming back with any more bad news like "Will jeepers Wally we lost all the data"

I don't know Beave Dad is gona be pretty mad"

Sure Wally!

[Freakindj]

Gee who would da thunk it Wally, Both drives get corrupted off 1 raid card.

I know Beaver technically it is possible and it sounds like a really good excuse. I sure hope Dad buys it.

In walks Eddy Haskle

Boy are you guys messin up. Tellin Ward that story is going to get you guys in so much trouble.

Ah Eddy what do you know, you didn't lose 10 days worth of data !!!!!

Gee Wally mabey he has got a point

[simonbg]

I would normally keep my mouth shut in this situation and not distract efforts away from getting it resolved, and I have no doubt you are personally working hard on this, but given the length of time this is taking, I am finding it hard to keep quiet.

Can we please have an update on progress? The last one was over four hours ago, and the backup that was mentioned previously should have finished some time ago, so can you let us know what else has gone wrong?

Thanks

[dscotts4]

Well, well, I sat at my office at 4:55 PM EST yesterday and waited for 3 hrs because the support staff at HG told me the server will be up any min. 3 hrs later I gave up and head for home. My customers understand that now and then there are computer problems and their sites may be down for an hour or so. I waited and waited, hoping that the system will be back up and all will be well.

Over 15 hrs later, sites still down and now we are told that you have to restore from a backup that is about 10 days old. What about this weekly offsite backup that is posted on your site under the hosting package. "Every server in out fleet has a Raid1 setup that provides instant backups. We also have a weekly offsite backup."

I see lies all over this, I have lost a few of my clients already and most of the rest will leave when I tell them about this 10 days old backup that we will have to load.

I know you guys are working hard to get the sites back online but the problems have already cost us the resellers more issues then we can deal with.

I know I will be looking at other hosts reseller packages. I have been here with hostgator for about 7 months. I notice my sites were down about 8 times and I'm only on the site about 5 hrs a week and I check them about 3 times a day (when I have time). 2 out of the 8 down time were down for over 1 hr and now this 1 is over 15 hrs and still going.

What I want to know is what steps were taken after the other server went down for so long to keep other server from having similar problems?

What step will be taken to try and gain our trust back?

Can raid be setup so a bad drive can be replace with the system still running?

Can a reseller have 1 reseller package but have it setup across 2 or more servers (so only a few client's will be down at a time)?

why is the weekly backup over 8 days old when it should be the most 6 to 7 days old?

Thanks
dscotts4

[kwe]

I was given the same story yesterday and also last week when I requested a restore on one of my resold domains. I was told a 10 day backup was the most current because the previous week's backup was "corrupted" and that the backup issue has been corrected. That is not what they are saying now.


I am really tired of the lies from this company. I am a network engineer and I am responsible for a world-wide government network so I know what I am talking about when I say that restoring a backup shouldn't take 14 hours (and counting) and that all backups should be verified. The problem with relying on a RAID1 (disk mirroring) solution is that there is no error checking so if bad data is written to one drive it is copied to the other. It uses only 2 drives and is the cheapest form of fault tolerance. By the way every computer is at least RAID0 (single drive with no redundancy). While certainly more expensive, RAID5 offer error checking and the ability to hot swap drives without any loss of performance. Also with a RAID5 you can insert a blank drive and it will be automatically updated so there is no downtime due to drive failure unless 1/3 of your RAID5 is corrupted.

[GatorBrent]

we believe should be done in 2 hours.

[GatorBrent]

We had to pull the box down, because the array rebuild was causing the restore to go painfully slow. Now that the box is running on 1 drive the restore is going very fast.


Raid1 is not a perfect solution neither is raid5. I spoke to one of our customers a few days ago who lost every drive in his dell raid5 instantly from some type of surge. That is why we also have the disksync backup which we paid over 10k for. Disksync = evault.com who his the third most known company for backup solutions. (my understanding)

Their solution does not work on reseller boxes correctly due to many bugs from file structure being too deep as well of other issues. We have been promised a fix for months every day being told "tomorrow"

it has not been a money issue... no one has been delivering for us!

We gave up on evault recently and are having our own solution developed. We have done some testing on it, and hope to have it completed in about a week. Our solutions will be pretty much the same thing as disksync but without all the bugs.
We know what we are doing however our let down a lot from others. Anytime something hardware related needs to be done theplanet usually screws it up many times before getting it right. They have a great network and support but if you ever need anything hardware related done you're in trouble unless you're lucky enough to get one of the two people there who actually know what they're doing. (hours later)
so going forward....
The chassis was swapped we were told, and everything should be good, once the data is restored.
We need a better backup solution which we are working on from developing our own.

We are talking that to them about mailing out a kvm switch over ip. This would allow them to plug in a piece of hardware to any downed server to give us access. This would be almost as good as us physically being there if any problems happen to the server. Currently we are able to fix almost anything unless we do not have ssh. When a server is off-line there is no ssh which means we are in their fumbling hands to get it fixed.
Nobody likes the current situation we are in with your server... we are not happy, your clients aren't happy, and you are not happy. Hardware does go bad, and when you have as many servers as us it is bound to happen. Everyone is upset, and everyone is being yelled at however no one feels it as much as we do. It is our job to get it fixed, we do not have the luxury of going to sleep knowing it will be taken care of.
we're doing everything we can.........
And we are extremely sorry with what has happened. Could we have prevented it? no...
could we have better backups? yes, but unfortunately we were sold a half working solution.

[Lumis]

So only 1 hour left then??
or 1 week?

[Lumis]

Wish I could wait and see the result, but amazingly after being awoken at 4am with the first sms from our sales rep (he in time zone 2 hours ahead), then again at 6am when he started panicking, it is now 11pm,
19 HOURS LATER............unless my maths suck after sleep deprivation....
I didn't take much notice at 4am, and I told him to don't ever wake me again cos Gator would sort it out. I doubt I'll ever take that attitude again, I have read with horror some of the other difficulties Gator has had with servers going down and The Planet to blame for taking out the wrong hard drive or whatever excuse, and now it has happen to the Audi server i am on.
Ahhhhh, how frustrating is this for us resellers, what are we supposed to tell our clients. I need some sleep, but why bother if I have to wake in a few hours to another day of complaints from all my clients on your server.
And to recommend you, I would quote from your website by agreeing with what you say, hosting companies all offer superior sales support, but what makes the difference is when you need support.....

[Axon]

Can someone explain to me what happended to the "weekly offsite backup"?!?!?! Why are we being quoted 10 days worth of lost data?!?!

[bshell]

Getting all pissed off only makes someone want to work less. Any good reseller should be making their own daily backups and/ or have a backup server in case this something like this happens. As cheap as the reseller hosting is here, any company should be able to afford to do so. Chill out, guys. **** happens.

[scottz]

Lets not get excited about data we may lose. Complaining about lost services at this point is not productive.

Hopefully HostGator will keep us updated with the status of the situation. Please hold off on complaints until the problem is resolved and we know what our loses are. HostGator can't handle our questions until the problem is resolved anyway.

-Scott

[GatorBrent]

45 gigs restored still has another 10 or so. It's on auto pilot so no way to speed it up. Last I checked it was doing a gig every 5 minutes or so.

[GatorBrent]

be done any minute now...

[Freakindj]

Brent your server status page said the audi server is back up again

http://hostgator.com/s/

however my frontend/x/status.html is indicating the apache failed

can you clarify on this

[GatorBrent]

The box is done and seems to be good accept for being flooded by everyone checking things. Any bugs?

[kwe]

What is being done to ensure that more current backups are kept? When I requested a site be restored from backup last week the most current you had was 2 weeks old and I was told that the error was corrected and that it would never happen again. Well, one week later here we are again and you still haven't corrected your backup process.