supra 1/2/05 major outage

[jooosh]

Yeah...he mentioned it would take 72 hours, IF IN FACT THEY WERE DOING A RESTORE. His point was that they were not doing a restore from backup otherwise the sites would have been down for 72 hours. At this point the only thing Brent has indicated is that they've restored the RAID not pulled from any backup. Once again though, I think the DC is just blowing smoke.

[airkat]

Originally Posted by jooosh

Yeah...he mentioned it would take 72 hours, IF IN FACT THEY WERE DOING A RESTORE. His point was that they were not doing a restore from backup otherwise the sites would have been down for 72 hours. At this point the only thing Brent has indicated is that they've restored the RAID not pulled from any backup. Once again though, I think the DC is just blowing smoke.

Which if is true.. this means they restored a backup, which didnt take 72 hours which = more bull****

[GatorBrent]

more bad news......

The raid controller stopped writing to the current drive about a month ago.
This is why everyone has outdated information because it is in fact a month old. looks like it failed nov 22nd to be exact.

Here is the chain of events........
the raid controller failed and stopped writing to the current drive a month ago. A few days ago the server’s primary drive had a hard drive failure.

Of course this isn't enough problems so here's the icing on the cake.....
We have backups from just a few days ago on the weekly offsite backup. We know this because we checked it a day ago however the server is now off-line from what we are guessing to be a hardware failure. We have two backup servers that handles backups for all our servers. Lucky supra user's backups are located on the one that went down after the whole supra fiasco. We do not have any reason to believe the backups are affected but we are not getting any response from tailormadeservers.com (who we go through for backups)

Hostgator has never had a problem this bad with a server. Every possible thing that could go wrong pretty much did.
With all these hardware failures it's almost as if the manufacturers have things programmed to break, right before the holiday season.

Were were at now...
we are working to get the issues fixed everyone is posting about. We can’t do anything about the data being outdated at this time. I will post when the backup server is back online and those who need fresh backups will be provided with them.

I will let everyone know when a meteor falls from space and hits supra. It is bound to happen with all the issues we are encountering.


I understand how upset everyone is and for those of you who are still understanding words cannot describe how amazing you are. For those who want us to suffer more take pleasure in knowing the last few days have been a few of the worst days of my life and fellow staff members.

I will continue to keep everyone informed.

[jooosh]

Thanks for doing all you can Brent. I don't blame you personally and I don't think anyone else should. I've been in your shoes. It's just been an unfortunate chain of events. Just keep doing your best to keep us all informed so we can make the proper decisions for our clients.

[jooosh]

On a note of moving forward: How stable is Supra now? Should we all be looking to do updates if we can in fact access the server? Are we at risk for any more loss of data at the moment? I.E. If I make any changes now, will a potential restore write over them? Just wanting to know if I can take any action at this time to get my sites updated.

Thanks

Oh yeah...and any idea of a time frame relating to your response would be appreciated.

[jooosh]

Not to change my tone, but concerning accountabiliy: Why didn't the DC know of the failure back in November? This concerns me considering the business that they're in.

[liquid]

Speaking for myself (and probably a few other resellers), all I really want right now, more than anything else, is to get into WHM and cPanel. Seriously.

[airkat]

I'm right with you. This is ridiculous, Fine, HG is dong all they can, but you cant expect me to take it and smile. How about handing out accounts on the other server as previously announced

[airkat]

I just got a PM from "ztechPC"

I do not have access to post and its pissing me off but maybe you can relay this for me.

If a raid controller fails the system will shutdown immediately because it will not be able to access the drives, if that was the case they would have known back in nov, they are just making bs excuses and insulting everyones intelligence.

[ghumphrey]

Brent,

Thank you for the update.

I have a few questions and as I am getting no response from support@hostgator.com right now (and I understand that you are probably swamped with email,) I thought I would ask them here so maybe someone else would be able to use the information as well instead of contacting support yet again.

1. If I was to bring the sites up on another server (I have another account @ HostGator on another server,) would this affect the possibility of getting a backup of the data from a few days ago?
2. Would setting up this account update all of the DNS servers in your group so that people looking at the old DNS will see the new server?
3. Is there an ETA on when we will get updated passwords, even for the main account? I cannot get in and get access to the information that is currently there...which is better than nothing.

Thank you (and the whole HostGator team) for your assistance.

[scooter1219]

All we really want to know is when all will be back up and how much of a refund we are to expect?

[Denoha]

Come on HostGator. I really did not want to find another hosting service but you are making my decision for me at this point.

[webmorpheus.com]

when will the email be up?

I just had a new mortgage bank with 30 email accounts and theIR loan officers ARE PISSED OFF!!

And they just switched to me because their old host had ****ty email service....

[GatorBrent]

Cpanel log in for end-users is being worked on right now. The server itself is very stable. Now that all the affected hardware has been replaced. There is a few broken things still such as cpanel log in that we are fixing one by one.

Please hold off from requesting and refunds at this time. If you could wait a week before requesting that would be great. If you wait a week, we will give you two months free. If you wish to have it now we will give you one month. We are overwhelmed with taking care of problems at the moment, and anything that can wait will help out.

[airkat]

Originally Posted by GatorBrent

Cpanel log in for end-users is being worked on right now. The server itself is very stable. Now that all the affected hardware has been replaced. There is a few broken things still such as cpanel log in that we are fixing one by one.

Does that include WHM?

[flipswitch]

Ditto, finding another host service is a royal pain. My sites are for the most part are up. None of my perls scripts work anymore. Email does not work and cannot use WHM or CPANEL.

Please get this thing fixed so that I don't have to move all my client sites, again.

[TheImmortal]

Hold off on Requesting what? Sorry if I am unclear. Just for my sanity and that of my customers, are backups available from 1/1 ? I finished putting a new site up on New Years eve, I edited all the HTML (Mambo) on the server as opposed to locally (in retrospect, I should have copied it locally as well). As long as I know that somewhere there are backups of all that work, I can manage the expectations of my customers and turn away their wrath.

Crap happens, HG; keep up the good work in recovering this mess. At least you all are on top of things, unlike the DC so it seems.

[ztechpc]

I already started transferring my sites.

My concern now is to get the passwords fixed so in the meantime peoples email will work and so I can figure out what I am going to do with mysql databases and ssl certificates.

[geoff]

Like we really have time to set up a login and password for this forum to complain but our sites are in terrible shape, all the updates we did in December are gone and customers with data bases full of customer information entered in December is gone too. email account passwords are gone and we cannot log into the main site to do anything - that is why I have time to post here. But that means my business is in the toilet. When will this be fixed?

[webmorpheus.com]

do we have to manually send a ticket to support@hostgator.com to reset our CPANEL pw's?

I dont have time for that!

[redlinemiami]

At this point, the damage is beyond repair. How and when can I access my data and all sql databases.

And I don't mean...1 month ago data...I mean as of this last Saturday.

[airkat]

I wonder if they'll reset any of my passes for me seeing how I cant email them from my normal email address....

[Matrix0]

What I want to hear now is:

a) Passwords for WHM will be enabled soon. We all need access to that ASAP.

b) Email systems/etc will be fixed shortly.

c) Confirmation that our VPS/etc systems are now stable and won't be changed

d) Access details/etc to the server/whatever, which holds our most recent data, will be provided in due course, so that we can restore ourselves ('due course' being shortly)

Please?

Obviously, downstream I will be looking to hear what steps have been taken to ensure nothing like this can happen again.

[jacob]

Sent a ticket for a password change in 2 hours ago. Still no response. At this point, I'm saying give me the backups, I'll take 24 hours of guaranteed downtime in exchange for a guarantee everything will be back when its over.

[redlinemiami]

I agree, I sent in a ticket about the same time...no response. Give us the backups.