we shutdown Mustang due to drive errors....

[GatorBrent]

The server mustang had a drive failure about an hour ago. We are currently working on having the drive swapped with a good drive. The good drive will then rebuild off the raid1 current good drive. This box should be back up and running as soon as this is completed.

[GatorBrent]

"
The server is currently fscking while rebuilding the array. Doing both at the same is a slow process, we are waiting for the fsck to finish so we can put the server back online and continue to rebuild the array while the server is online.

You will be updated as more information is available. "

[GatorBrent]

still fsking atmost these take 2 hours from what I've seen.

[megadj]

my customers, are pressure on me, and want the money in back, want to know, how much time, to adjust this problem? and this goes to cause some loss of data? Please

www.megadj.com.br

[GatorBrent]

We won't know anything for sure until the fsck is done. This could be finished any minute or take a hour. There's really no way to track this and no way to speed it up.


If fsck finishes and it comes up there will be no data loss.

[GatorBrent]

still fscking

I'm more impatient here than anyone, trust me.

[davoice]

Is there an ETA yet?

[GatorBrent]

Fsck is done but may need another it's stuck at /var

Were commenting this partition out to get box online and see what we have, than will fix /var

[GatorBrent]

will know eta when we get ssh access.

[megadj]

I am worried, pass + 5 hours and nothing.

I am losing money, customers complain very and very.

all the stopped servers, sites of the government and store without connection,

please

By MegaDJ Plan ... Copper

[GatorBrent]

it pinged for a min but ssh hasnt come up yet. Dc working on it.

[GatorBrent]

/usr partition is getting fscked before we can get in.

[GatorBrent]

The system is still running fsck on /usr

It does display the following message about every 5 minutes.

Error reading block 720931(attempt to read block from file system resulted in short read) while doing inode scan. Ignore error? yes

Force rewrite? yes

The block # increases for each line output.


this means it was never fskced first time.

[megadj]

hi, you, has forecast to come back to function the server…, still today necessary to give to a reply my customers, I am crying, and worried, therefore the damage is very great here

The fsck is still currently running very slowly, we will keep you updated.

[ibasq]

Are you going to do ANYTHING about the money that some of us may have lost in advertising due to our sites being down????????

I have already wasted $40 myself on adwords today. over 100 people clicking to nothing!!!! And that's not even to mention the SALES I have lost! ARGH!

I want to be reimbursed for it by you. Or give me a couple of months credit for my hosting or something!

Regina

[GatorBrent]

We found out what is taking so long. The 2nd drive in the raid which replaced the failed old primary is writing at 112kb/ sec compared to 30mbs (normal)

They are checking a few things out waiting back for a call.

[Hosting by Design]

Thank you, Brent.

I would just like to put my two cents in. Before we get too angry about the down time and what it might cost us ... let's take a deep breath and realize that HostGator is sweating bullets over this.

Some things just can't be solved immediately. I am just happy to know that the problem is being addressed. Imagine the alternative ... the problem doesn't get addressed and EVERYBODY loses LOTS of money.

Best Regards,
David Peake

[bonzihostin]

Well I am hosted on mustang too. Its bad news when your a reseller web host and people can't connect to sign up for hosting. Sign ups for me have been slow but typically I have someone interested in hosting and now I look stupid.

I understand that hard drives break etc its a fact of life just for me and I am sure everyone else its just really bad news.

[GatorBrent]

"(bsteward-04/30/2005 16:23:38):
Alright, as per our phone conversation, I decided to check the drive that was removed and marked as bad. It appears I was right in that the wrong drive was removed. When I first booted the drive solo, it booted up to replaying the journal on the drive, then to a command prompt since I requested single user mode. It replayed the journals without error, but I am doing further checks on the drive. I also tested the performance of the drive and found it to be on par where it should be. The drive is currently running a forced filesystem check, much faster than the other drive.

Once the fsck is done and I have the server back online, I'll give you a call with more information"

We may be up very soon now that they discovered the wrong drive was replaced when we clearly told them which one was the problem. A day of downtime cortousy of theplanet. I'm on the phone yelling will update as I get it.

[Hosting by Design]

I agree there is nothing positive in this ... but I just felt it was important not to yell at Brent and Co.

I have clients who don't understand why the server is down. It is possible I could lose a client over an outage like this. It is possible I could die in a car accident tonight. Life happens.

David

[GatorBrent]

Everything is up and running with no data loss at all. The server is a little slow with playing catch up from mail etc. It should be completely back to normal tomorrow.


here's a quick recap...........


We told the datacenter exactly which drive to replace. I myself spoke with multiple people on the phone making sure the correct drive would be replaced. They assured me the right one would be replaced, but they still screwed up!

The good drive was replaced leaving the faulty one that ran 1/100th of normal write speed.

This made a normal fsck take much much longer.

I called in many times trying to find out how and why it was taking so long. I even asked again if they replaced the correct drive. Finally someone agreed with me something was up they tested and swapped for the correct drive, and here we are.

Entire day of downtime on a 25 minute procedure? why because the planet hired someone who doesn't know difference between drive0 and 1
As far as credits please email sales.

[bonzihostin]

Well, thanks for the update. I will be emailing for credit and see how we go with that. Mainly because although I don't have many clients I am getting some grief from people and I am having to explain what happend.

I am not mad at hostgator they have always delivered as far as I am concerned but this incident makes us both look like the bad guys.

It's one of those things and I understand that but its the clients of ours who don't see it that way! Afterall how many other hosts are out there? And what do I tell them? The guy fixing your hard drive can't tell the difference between drive a and drive b?

To be honest if there was a more plausable reason behind it like I don't know the plant had issues etc etc everyone else was in the same situation blah blah but they weren't.

I have had less issues of late from a "cheap" reseller account I am trying right now than my own hostgator server and that's kind of annoying. Oh well, nobody died but I still have customers to answer to.

Thanks for the update its appreciated. One final question though, you said everything was restored etc but I have some forums on my reseller account that are showing old posts and not was was the "current" ones before. Any reason why?

[123456qwerty]

So what exactly is happening to the email, website updates, databases from Tuesday up till now??

All my email between today and last tuesday is gone!

[GatorBrent]

Checking into anything missing now. this is straight from the raid1 so it should be exactly the same unless the raid stopped working days ago.