supra 1/2/05 major outage

[liquid]

Originally Posted by GatorBrent

I repeat, nothing has been restored so yes, all the data is current. Up to the time of the failure. Many passwords have been lost, as stated above.

I just had a phone call from a client, saying that they had made a bunch of changes in a mysql database just before Christmas, and those changes are all gone now. How is that possible if nothing was restored?

[ghumphrey]

I just checked one of my customer's sites and it is running from VERY OLD data.

Any status on when it will be running recent data?

[Kevin Moreland]

Exim configuration error in line 781 of /etc/exim.conf:

[airkat]

Originally Posted by GatorBrent

I repeat, nothing has been restored so yes, all the data is current. Up to the time of the failure. Many passwords have been lost, as stated above.

Brent, all of my sites are running VERY old content.

[airkat]

Also same as others on logging into whm.

When I user 67.18.2.226/cpanel/ it tries to log in then hits an infinite redirect loop until my browser stops it.

[ghumphrey]

My content is definately from before 12/5/2004.

I can't tell exactly because I am not able to log in yet.

[jooosh]

My guess at this point is that Brent was misinformed by the DC that they recovered the RAID. Data would not be this old if the RAID was recovered.

[GatorBrent]

What are you saying is old? database? or static pages? Can you name a page that's old?

[ghumphrey]

Old Static Pages

http://www.PlusVillage.com

That page was from before Thanksgiving.

Old Database

Users from November, no users from December. No Orders. This was from just before launch, sometime in Mid to Late November.

[fvsoftware]

Brent...
I have all the backup of my site, can i switch to another server with my dns??

[Denoha]

Here is a prime example....

http://cedarwoodtraders.com/catalog/

I made several changes to the website in December, removed categories, changed the header graphic, added categories and approximately 100 items.... all of it is gone. The last changes were made approximately 12/28/04 when I backed up the database, that too is gone.

I'm also still experiencing mysqyl errors, such as seen on this product description page...

http://cedarwoodtraders.com/catalog/...products_id=93

Warning: mysql_query(): Unable to save result set in /home/cwtrader/public_html/catalog/includes/functions/database.php on line 44
1030 - Got error 127 from table handler

select count(*) as count from reviews where products_id = '93'

[TEP STOP]

And also still no access to WHM so I will email support now.

[webmorpheus.com]

use a new Kernel, 2.6.9

[webmorpheus.com]

Alright Were Up Baby!! Hg4life

[airkat]

Originally Posted by webmorpheus.com

Alright Were Up Baby!! Hg4life

Maybe YOU are...

Content is all very old. Looking at pages I CAN access? We're talkign early december if not before then. Like someone else suggested, maybe the datacenter misinformed you...

[jooosh]

I developed a site on Jan 1st up 'till midnight (day before outage). It doesn't even exist now.

http://www.harvestcolorado.org

It worked on Jan 2nd 2am MST, and then was gone when things went down. This is how I know that they didn't recover the RAID and they must have restored from a backup.

Real bummer too 'cause it was being announed on Suday for people to sign up on.

[Denoha]

I had to make multiple changes on this site due a court decision in early Dacember and really need to get those changes back asap, but can't access anything due to the password issue.

http://www.kyrescue.com/

Waiting for an answer back from support via email. This is very frustrating.

[Denoha]

http://www.soulsticesiberians.com

Has three updates missing, first on 11/30/04 when a tree crushed the kennel and pictures were uploaded, another the following week, and another recently on 12/31/04 with general updates and cleanup.

[Trinity31]

Some of my domains still are not working... as in they don't resolve at all.
trinity31.net, nbserve.ca, rippedcanadians.ca, and others.

I also can't access my WHM/CPANEL or any of my email accounts.
Also, PHP isn't working either.

Please give an ETA!

[Reliquary]

Same problems as everyone else:
No passwords work, not even older ones in whm nor client cpanels.
Email not up and running
Data is quite old; i have an active forum on one of the sites and the newest post is from the first part of december.

[airkat]

Hope this is enough proof to support that its not an isolated incident and data WAS lost. I dunno if they did a restore or what, but this is NOT recent data.

I can uplaod recent content, but cant even access my friggin sites

[Denoha]

Same here Airkat, I have most of the changes just sitting here waiting to go but can't access anything.

[jooosh]

Another thing to note: Even though I can't access my email account, due to the lack of a current password, mail is still being bounced. Just sent an email to my account on Supra and no go. It was bounced back to my Gmail account.

[Trinity31]

I've got a page telling me the date is Nov 17... I pray to god this is not so!
Everyone else is saying that it looks like old data was restored, if this is the case then I'm going to explode, especially if it is as old as November!

Also, some pages on my sites are totally messed up!!! Some pages work, other pages bring up some sort of corupt visitor log page!

Example: http://www.vacanada.org/staff.shtml

[airkat]

I suspect mail has been bouncing since the server went down. A number of people reported bounced emails... Thank god for gmail



At this point I dont see why it wouldnt be easy to just restore the last weekly backup ( assuming you guys actually have one ) and chick the data there since the raid obviously didnt rebuild right.... Or just give us access to restore our data... OR accounts on a new server where we one can upload and begin again for the people who are staying with HG after this.

[ghumphrey]

Airkat,

Brent did mention that it would take apx. 72 hours to restore [from the last backup].

I was thinking that they should have been doing a restore to another server while they were trying to fix this one and let the server that was done first take over. Then if it was the restore, we would get access to the older server to grab anything that was usable from between the last backup and the restore.

However, I just sit here and listen to a ton of voicemail and reply to a ton of emails from my customers.


[Note: Edit added clarification of "from the last backup", Brent didn't give an ETA for the recovery from the RAID fault.]