supra 1/2/05 major outage

[GatorBrent]

It's still resyncing. We want this done before the day starts tomorrow just as much as anyone here.

Recent problems on supra are a result of the raid controller failure. Any one that wants to be moved to a different server is welcome to signup from our site however with the amount of time dns takes and all downtime would most likely be longer than waiting this out while we get it backup and running.

This should be done any minute now. (or few hours)

and we will than no how to proceed. We believe even if this doens't work we can hack some way to access data rather than resort to older backups.

[airkat]

Brent: DNS wouldnt take longer than this has taken. Ive gotten DNS fixed in a matter of hours. In fact, I had to point one of my sites to another host in the meantime just so I can get emails again. And DNS is fully resolved and all.

[Brink]

I can't speak for everone who is on Supra but I would think the weekly backup would be sufficient. Thats all we were promised. Drive mirroring is great... when it works, But we were told that there were weekly off site backups being preformed.

If you guys could just toss in a couple of new drives, format them & restore last weeks backup that would work great for me.

[airkat]

Problem with that is I JUST launched a site new years day. We bought traffic and all to it. So we're out money on several fronts here. Luckily I thought to store the media files on a different server with all the HG issues lately. So if they restore from last week, I'll need to recreate and re-upload the whole site wont I?

[fvsoftware]

I would like a new account on a new server. I have my own backups. What I have to do?

[GatorBrent]

I have been waking up every 30 - 60 min to call and check on status. It is done syncing and I was told a tech is ssh in for the first time since it went down. This is great news since we coudln't ssh in before.

I didn't get many details since he was on the floor working on it he should be updating me soon. If the data is still there with ssh access being obtained lyron can fix things as long as he has access.

[fvsoftware]

Brent could you set a dns redirect? I have another server, if this is possibile could be more fast then re-set all my dns...

[GatorBrent]

I just found out the hard drive went for sure and possibly the raid controller as well.

The main drive out of the 2 was toasted from going bad but before it died it caused data incosistancy / corruption on the 2nd drive messing up password files + 1000 other things we'll prob have to fix no one will notice.

This left the 2nd drive needing a fsck. You know when you shut down windows without shutting down? Pull the plug etc.. and when you boot backup it runs that check? It's like that. The check fixes the drive otherwise it won't boot because of being incomplete. Once the array was done / rebuilding the tech was able to hack in and find a backup pw file we had on the server / copy it over. Once this was done it gave the ssh access.

That leaves us with..
home having filesystem errors. with illegal blocks.

The above can be fixed from a fsck which usually takes about a hour and is being ran now. This should put us back in business in about a hours time.

This is the best news we've had since it went down!

[GatorBrent]

There's no way to redirect since the server that's having the problems contains everything including zones which can't be accessed.

Give it a hour.

[GatorBrent]

still running should be done any minute.

[fvsoftware]

we are waiting...

[onsite]

Well he did not say which minute. It could be any minute like he said. 1 minute or 50000 minutes. It's really nice having to depend on people who do this as a hobby and they get paid for it.

Let's see if we can retrace the events. The raid went down. They had to get a guy to scale a 16 foot high fence. Why he had to do that to get to his server is beyond me. One would think that they had a KEY but who knows. Since it takes about 5 minutes to replace a raid controller I don't know why they did not have a spare there that match what they were using instead of having to waste time and get the right one they should have had the RIGHT ONE IN PLACE. Then they had to respawn the data from the corrupt drives that should not have been corrupted on a mirror. It's 45 gigs. IT takes over what 17 hours to do that? IT would have been better to restore from backup to get as many people up as possibel then get the data from the drives. With DVD backup that would have only been about 7-10 DVD's which would not have taken a whole day to do.

These kids are playing with people money and they are not responsible enough yet. They keep having more and more downtime as they sell more and more people while their service goes to pot.

Very disapointing.

Originally Posted by midianet

...actually quiete a while already...

[GatorBrent]

It finished the fsck a few minutes ago, and upon reboot it gave files in /lost+found have an incorrect filetype. This means that the home partition is done fscking, but another partition needs to be fsck'ed

this one is not 150+gigs so should go much quicker.

onsite we are code located in the planet datacenter which is one of the largest data centers in the world. I had no idea why they did not have a key. Calling us kids, because of a hard drive failure is not going to speed the process up. Perhaps dell, ibm, and compaq are all kids as well since their servers and computers also have hard drive failures. Just because it's 45 gigs it still has to write 0's and 1's for the entire 200 gig hard drive. It's an automatic process which we can't play a part in even if we wanted to once the array rebuild starts.

Let's try to focus on the problem, which is being worked on.

[airkat]

No I think he said it ebcause this could have been solved by now. I cant even entertain offers from any other hosts because I HAVE NO DATA TO TRANSFER!!! Soon it will be 2 days without the site up. Thats what ~92% uptime for the month... on the third day of the month.

I dont see why we couldnt get a temp server up with the week-old backup. Then the people who stated they either were leaving or wanted a new account on another server could do so....

[MachineX]

I totally agree with you Brent.

I've been down like everybody else in here... I've been looking this forum in silence since yesterday. I understand the problems each of you are getting from your customers since I have them too... But everybody knows it's computer stuff, and this kind of things happen some times, it's actually normal.

Of course, I agree it's been a bit long !! I guess were up for a free month

[onsite]

They don't stay down for days. This is the second day they we are down. they have the correct parts on hand to repair problem when they happen. They don't have to scale a 16 fence to get to their own equipment. They don't tell you it's going to be minutes when they know its going to be hours. You said at 6:54 est that it's any minute now over 2 hours later still no sites. They listen to their paying clients and accept blame when it's their fault. they are professional. Can I tell my clients that your guy had to climb a 16 foot fence to get to his equipmet then replace faulty equipment with the wrong parts? How does that make me look? What you do reflects on each and every customer you have not just you.

I would say that you are running a Mickey Mouse outfit but MM knows his stuff and gets things done. I've very upset because my people are calling me and I can't tell them anything because I don't have faith in what you are saying because each time you say something it has not happened. What do you want from your customers when you promise something and don't deliver. You are costing me money and time. Time that I have to spend on this is time I can't spend it on working.

Originally Posted by GatorBrent

It finished the fsck a few minutes ago, and upon reboot it gave files in /lost+found have an incorrect filetype. This means that the home partition is done fscking, but another partition needs to be fsck'ed

this one is not 150+gigs so should go much quicker.

onsite we are code located in the planet datacenter which is one of the largest data centers in the world. I had no idea why they did not have a key. Calling us kids, because of a hard drive failure is not going to speed the process up. Perhaps dell, ibm, and compaq are all kids as well since their servers and computers also have hard drive failures. Just because it's 45 gigs it still has to write 0's and 1's for the entire 200 gig hard drive. It's an automatic process which we can't play a part in even if we wanted to once the array rebuild starts.

Let's try to focus on the problem, which is being worked on.

[GatorBrent]

Server came online errored with Starting sshd:/usr/sbin/sshd: error while loading shared libraries: libKrb5.so.3: cannot opeb shared object file: no such file or directory
.
.
.

Starting httpd: /etc/rc3.d/S85httpd: /etc/rc3.d/S85httpd: cannot execute binary file
.
.
.
Starting postgresql service: su: user postgres does not exist
.
.
.
Starting cPanel Log services: execvp: no such file or directory



We are having dc fix ssh once that's fixed we can fix everything else ourselves quickly.

[fvsoftware]

I hope nobody lost some data....

[airkat]

And here comes the calls... *sigh*


...I also dont see the point of the fence story... does that mean he stole a raid controller?

[vbtweb]

Please get it up Mine has been down since Saturday. I'm tired of being yelled at

[GatorBrent]

ssh now works and we have access to the server for the first time since the hard drive failure!!! We finally have a chance to fix this from our side.

[GatorBrent]

User content including mysql appears to be good. With all the corruption this is one of the things we worried about with getting this far. At this moment apache along with many other services that were corrupted are being reinstalled.

[airkat]

With that being said.... how. much. longer....?

[Brink]

9:59am

I'm back up.

Much thanks to all who worked so hard.

And Happy New Year!!!!

[airkat]

I assume they used an old backup? None of my crap is there