Lotus Server

***GatorJosh*** · #1 10-10-2006, 02:44 PM

We are seeing an issue with hardware on this server causing downtime. The Data Center is currently working to swap the bad hardware and bring the server online as soon as possible. This will not cause any data loss.

We will keep this thread updated as we get more information.

***GatorBrent*** · #2 10-10-2006, 02:56 PM

Were having them do a chassi swap trying one drive at a time to see what the status of both of the drives in the array is.

***GatorBrent*** · #3 10-10-2006, 03:45 PM

It's failing to boot with either drive and new chassi. We're trying to figure out whats happening with it. One server has a live cd being put in, and another has a fsck running.

#4 10-10-2006, 04:12 PM

thank you for keeping us up to date. we have many clients very upset with this down time.

#5 10-10-2006, 05:18 PM

waiting it out is very painful - so much for 99.9% uptime
Most customers do not want to hear about why. They just want to know there is full redunancy. Apparently this is not the case on this server...
How can this be?

***GatorBrent*** · #6 10-10-2006, 05:18 PM

The data looks fine on the drive however we are getting some bad blocks on doing sys check on boot.

#7 10-10-2006, 05:47 PM

Can HG give us an ETA??

***GatorBrent*** · #8 10-10-2006, 06:05 PM

We're trying a few things still to avoid having to restore from backups which would be at least 20 hours of downtime. We should know soon, as you can imagine we don't want to have to restore from backups as it's a slow process without any way to speed it up.

#9 10-10-2006, 06:16 PM

This is getting ridiculous!
It's been over 8 hours (that I know of)

***GatorBrent*** · #10 10-10-2006, 06:36 PM

It hasn't been 8 hours it's going on 5 but yes it is rediculous and we are currently restoring from backups since both drives failed simultaneously on the raid1 that it was setup on.

There's no way to know how long restoring from backups will take, but it's an automated process that usually takes about a day getting some people up within minutes and others up 24 hours or more later. I'll see if we can figure out a better estimate.

#11 10-10-2006, 06:37 PM

What kind of "make good" will you do to keep us on as clients. Unless we have some kind of free year or something of hosting I am taking all my host gator accounts some place els.

#12 10-10-2006, 06:40 PM

My records show it going offiline for me 8:32 am central time. you can do the math.

#13 10-10-2006, 06:42 PM

I loged in at 10 and had nothing. I have clients that report is being offline earlyer.

quietFinn · #14 10-10-2006, 06:47 PM

Oh I can't wait... soon some other "unregistered" says it was already down 3 days ago

#15 10-10-2006, 06:54 PM

It has been down for 3 days..... Joking. the fact of the matter is that we have a down. if nothing els will work we need to all Pray. only God can help a gator in a data center in texas. speaking of texas.... texas did win this weekend. go texas. if only there gators would host

#16 10-10-2006, 07:08 PM

Quote:

Originally Posted by GatorBrent

It hasn't been 8 hours it's going on 5

I'm going on a customer that call me at 12pm telling me that he has been trying to send email since 8am without success and since it is or was about 1/2 hr ago it would be 7 and half hours. But I’m just going by what I was told!

Quote:

but yes it is rediculous and we are currently restoring from backups since both drives failed simultaneously on the raid1 that it was setup on.

Now what are the chances of both drives failing at the same time!

#17 10-10-2006, 09:30 PM

any news?? webs are on-line, but cpanel is down "Invalid license", and emails account give this error "-ERR Maildir invalid (no 'cur' directory)"

***GatorBrent*** · #18 10-10-2006, 10:19 PM

Sorry for the lack of updates. I went to update and my pc crashed and got sidetracked shortly after.

Dave managed to get the server up and running a while ago. He did an rsync bringing over system files from a working box to get it up and running.

What happened?

There's a bunch of hardware related possibilities so here we go....

1. One drive failed causing corruption on both.
2. one drive failed and one was failing causing corruption on both.
3. Both drives failed from something hardware and while it's up it's struggling to live.
4. 3ware failed causing corruption.

5. million other possibilities...

We aren't to sure what happened but the drives are now testing bad and there was a lot of corruption to system files / partitions such as /etc booth etc.

The box has been up and running fine for a while, but we honestly don't trust it so our syncing it to another box to do a full chassi swap.

If we had continued to restore from backups downtime probably would of exceeded 24 hours so this was a pretty good outcome so far. I'll keep you updated when the chassi swap will take place. (20 minutes of downtime after we finish this rsync to another server)

#19 10-11-2006, 11:54 AM

So what is happening today again?
Talking about getting ridiculous!

#20 10-11-2006, 12:01 PM

Any eta on getting a valid license file for cpanel?

#21 10-11-2006, 01:21 PM

Quote:

Originally Posted by GatorBrent

Sorry for the lack of updates. I went to update and my pc crashed.

Now what are the chances of that also?!

Sites were down again today!
What is up with this "Invalid License File".
Can't access cpanel or whm panel.
About an update so we may make plan on going back to work.
I need to access my database!!!!!! (yesterday)

kmaw · #22 10-11-2006, 08:13 PM

Wow, the many posts from 'unregistered' would worry me...

#23 10-13-2006, 07:49 AM

WHAT IS HAPPENING?!?!?! now i lost 8 account from my reseller acount, i has 86 domains and now ONLY 78!!!!!

***GatorBrent*** · #24 10-13-2006, 04:57 PM

This box has been chassi swappd and is no longer degraded so everything is back to normal on it. In order to do a rsync you have to copy all the data from one server to another. There's so much data any time you do a transfer like this it can take 1-2 days. So what happens is we transfer all the data including mysql, and then at the time it finishes we do just the mysql again to make it as fresh as possible before doing the chassi swap. This usually results in about mysql being 20 minutes old on the new chassi and static data a day old.

If you created accounts or did file updates during the time everything was being moved over they would no longer exist since all static data would be about a day old and only the mysql would be fresh.

Please pm me if you have any questions. Thanks!

Serra · #25 10-13-2006, 06:11 PM

Quote:

Originally Posted by Alberto

WHAT IS HAPPENING?!?!?! now i lost 8 account from my reseller acount, i has 86 domains and now ONLY 78!!!!!

You have 86 accounts on a shared server? Time to move up to a dedicated server. So many fewer problems.