supra 1/2/05 major outage

[liquid]

Sites are up, but I can't get into WHM yet. I assume it's still in progress.

[airkat]

Originally Posted by midianet

looks like we are back on

Not entirely... and this backup honestly looks way older than one week? And there does seem to be data corruption....

[airkat]

Originally Posted by midianet

have my sites back but none of my passwords work anymore...not for WHM, not for cpanel, not for webmail...help?

Relax.. I dont think they're done yet. But again, this backup looks ancient.

[Brink]

FTP allows my old password. I changed that on 12-24.
Also host not found on a domain that I installed and had working on the 23rd.

Looks like the backup is from 11-22-04

So, it looks like we all have a lot of work to catchup to where we were.

[airkat]

Atleast u can get in

11-22 or 12-22

[Rendus]

Brent, in the future I might advise against giving these people updates on what's going on. Leave them in the dark, let them not have the slightest idea what's going on, and let them rot. Mysteriously fix the problem, but restore a month-old backup and not answer any questions about it.

Seems that's what they want, rather than a play-by-play on what's going on, and why...

[Brink]

11-22 is the date on my inbox file in the mail folder.

That gets mail every day. So I'm assuming that is the date of the backup.

FTP should let you in though. Try old passwords if you change very often/

[airkat]

Originally Posted by Rendus

Brent, in the future I might advise against giving these people updates on what's going on. Leave them in the dark, let them not have the slightest idea what's going on, and let them rot. Mysteriously fix the problem, but restore a month-old backup and not answer any questions about it.

Seems that's what they want, rather than a play-by-play on what's going on, and why...

And how did you come to this conclusion? You think we are wrong for being upset over a 1.5 day outage? And if the stuff im seeing on my sites are any indication then YES he DID restore a month old backup... older than a month in fact.

*edit* and in this particular play by play its been revealed stuff has happened that shouldnt have... why should someone need to scale a fence

Why is the backup so old? Why couldnt we pop new drives in the interim until the old ones were recovered? Do YOU have answers to this?

[fvsoftware]

i think this backup is one mouth older....

[Brink]

Actually Rendus,

We appreiciate the play by play. Or at least I do. It was great that we got updates often enough to show that someone cares that we are down.

You did a great job Brent! (although you could have left out the part about scaling fences That kinda makes it sound a little bogas even if it's true.)

[fvsoftware]

my WHM password is changed... i can not log in. I'm sure.....

why??

[GatorBrent]

Nothing has been backed up / restored. It's the actual servers hard drive. Most people seem to be up but their are still a lot of problems from the corruption we are curently working on. Will keep everyone posted...

[Trinity31]

I give HG credit... I'd like to see you all find another host who gives you this kind of update information!

Just as none of us have hands on access to the server... neither does HG, so unless you want to critizes yourselfs as well, then there's not much validity in critizising HG! I think the Data Center is most at fault here... they should have had this fixed long ago, if not prevented it in the first place. If the tech is having a hard time fixing it, then they should send 2 techs, or 3... or the whole staff!!

I hope HG will give them a good slap over this...

[airkat]

Originally Posted by Trinity31

I give HG credit... I'd like to see you all find another host who gives you this kind of update information!

Just as none of us have hands on access to the server... neither does HG, so unless you want to critizes yourselfs as well, then there's not much validity in critizising HG! I think the Data Center is most at fault here... they should have had this fixed long ago, if not prevented it in the first place. If the tech is having a hard time fixing it, then they should send 2 techs, or 3... or the whole staff!!

I hope HG will give them a good slap over this...

True enough.. I see your point. But we are all ( atleast I know I am ) doing business with HG and not the data center... So we don't address the DC with our issues, we address HG correct?

I just want it all over

[fvsoftware]

All my site still down... what about you?

[airkat]

ditto. I've just moved on at this point.

[liquid]

I can see my sites, but I still can't get into WHM at all. Now I'm getting an "Internal Server Error" instead of WHM not accepting my password. I'm not sure if that's a good or bad thing at this point.

[Denoha]

I sincerely hope Supra is still being worked on... some of my client sites are back up but there is no access to emails, cpanel or webhost manager and my sites that use msql have MANY errors and missing data.

[GatorBrent]

As stated we are still fixing some final corruptions. We had to take down apache to rebuild the account list to get the remaining people up and running again. This will be done in a few minutes.

The datacenter actually did an amazing job guys, most datacenters would have given up on our directions long ago / ordered a restore. The only thing I blame them for was about 2 hours lost from the raid controller being locked up and someone having to climb the cage. Them not having a key 24/7 is ridiculous!

When you signup for a host you are also signing up for the DC they use. If a host is in a bad DC no matter how good that host is they are going to be as good as the DC. We are in one of the best dc's in the country.

The huge time frame wasn't anyones fault. The server was doing the data transfer onto the raid array. This couldn't be sped up nor interrupted, and until this automatic process was completed nothing could be done.

With the amount of files on this server I believe it would have taken around 72 hours for the accounts to copy over from the weekly backup. Believe it or not I don't think it could have been done any faster other than the raid lockup chaos.


Almost the entire downtime was simply waiting on the array rebuild / fscking which couldn't be sped up since it's done auto by the server itself.

[GatorBrent]

All sites seem to be up and running now.

some problems being worked on still...
ip/~username/ isn't working yet.

Many users login passwords were lost and or reverted to something older. This had something to do with part of the corruption and an older file having to replace the corrupted one for us to gain access. The fix... if it's your main account email support giving them a pw to change your main reseller account to. If it's your client simply login to your whm and reset it using the password modification.

Anyone noticing problems not covered?

[airkat]

I remember you stating no backup/restore had been done yet. Is this still the case? I'm seeing VERY outdated content

*edit*

Issues:
http://www.idleriot.com/cpanel/
Server error

http://www.netkat.net/whm
same thing

[jooosh]

If I'm reading the thread correctly, I thought the DC was doing their best to recover the RAID itself, not restore from a previous backup. So wouldn't the data be as recent as when the RAID failed?

[Kevin Moreland]

Issues, yes:

Access permissions, or Perl not running. No contact forms work.

http://www.themusicdifference.com/
http://www.kykev.com

Also spoke with gatorsupport3 who was supposed to set me up a new WHM on a different server...waiting for welcome e-mail now. Trying to jump ship and stay with HG, but you're not helping.

[GatorBrent]

I repeat, nothing has been restored so yes, all the data is current. Up to the time of the failure. Many passwords have been lost, as stated above.

[GatorBrent]

use 67.18.2.226/cpanel 67.18.2.226/whm for the time being. If your login doesn't work e-mail support to have your password updated. I'm having someone check out perl now.