Echo Outage

***GatorJosh*** · #1 08-09-2007, 02:49 PM

The server has been taken offline for emergency maintenance. There is a degraded array that is causing the outage. We are working to get this server back online as soon as possible and will continue to update this thread.

esl · #2 08-09-2007, 02:51 PM

Thanks for the notice and updates.

***GatorDaveC*** · #3 08-09-2007, 03:38 PM

We are currently getting the server booted up to a LiveCD for further investigation. Currently the server will not boot even with a broken array (Removing one of the drives from the array).

We will keep you informed once we have more information.

jnjewlz · #4 08-09-2007, 04:03 PM

Grrrrrrr. My clients are freaking out. Thanks for the update.

blbrocks · #5 08-09-2007, 04:26 PM

My clients are calling with bad undertones... please hurry thanks! I know you are working hard to resolve...

blbrocks

***GatorJosh*** · #6 08-09-2007, 04:31 PM

We are currently fscking the partitions now. We are trying to get this box online ASAP.

We will keep you updated.

***GatorDaveC*** · #7 08-09-2007, 05:51 PM

The server has been online for about 30 minutes now. The downtime shouldn't have been this long. The data on the P3 drive (The good drive) seems to not have the correct array information.

The P2 drive (the failing drive) is currently the drive online right now. I have file system checked /, /usr, /var, /tmp, and recreated swap. I started a file system check on the /home, but it looks pretty clean, plus we are going to rsync the data to a new Conroe RAID10 server.

I'll keep you posted on the progress of the server migration, and anything else that may come up during the rsync.

ryduh · #8 08-09-2007, 07:12 PM

I can access echo.websitewelcome.com but not my domains. I hope that's how you guys have it temporarily set up.

blbrocks · #9 08-09-2007, 07:13 PM

Not sure I understand... you say something is online but my clients websites and email are still down. Sorry, not that server savvy... do you have a eta?

Sam · #10 08-09-2007, 07:24 PM

Quote:

Originally Posted by blbrocks

Not sure I understand... you say something is online but my clients websites and email are still down. Sorry, not that server savvy... do you have a eta?

They might be swiching the servers.. or apache has failed..

***GatorDaveC*** · #11 08-09-2007, 07:51 PM

No sorry, there was a problem with the partition that the zones were on. I had to fix that up and reload the zones in order for the domains to work correctly. Sorry about that.

All sites should be online now, we may need to add some extra restrictions to take down the load on the server for a faster rsync.

chaoren · #12 08-09-2007, 10:56 PM

Pls let us know in advance if there is going to be any possible downtime or interrruption or slowness to our sites during the server migration. THanks.

ryduh · #13 08-10-2007, 01:52 PM

Any updates on when the load is going to be back to normal?

13:52:04 up 20:32, 4 users, load average: 56.54, 51.86, 39.90

ryduh · #14 08-10-2007, 01:55 PM

Now it seems like they are down again. :[

#15 08-10-2007, 02:00 PM

yep, we are still having issues too. site is slow, or doesnt load at all

***GatorDaveC*** · #16 08-10-2007, 02:14 PM

Sorry about some of those outages the load on the server was extremely high at some points due to user resources. We have had to disable a couple of clients scripts to get the server functioning better.

The data has been moved to the new server, and we are in the process of bringing the new server online. Many of the home directories we have had to take from backup due to the constant IO on the server. Everyones mail and MySQL is being kept in sync to avoid lost mail and MySQL entries.

Once the new server is online, we will put up the old server on another IP address, so any data can be pulled from there.

#17 08-10-2007, 05:30 PM

again??? what is happening now? Why are we still having problems

ryduh · #18 08-10-2007, 05:33 PM

I am getting 404 errors on all my sites right now... interesting... :[

#19 08-10-2007, 05:47 PM

Quote:

Originally Posted by GatorDaveC

Everyones mail and MySQL is being kept in sync to avoid lost mail and MySQL entries.

I guess that didnt work. I just lost MySQL entries from the last couple hours.

esl · #20 08-10-2007, 06:06 PM

Patience.. it takes time to fix this.

***GatorDaveC*** · #21 08-10-2007, 06:06 PM

I did the best as I could. The servers were on different routers and it took about 2 hours to get the blocks of IPs to the other server. During that time the new server was down. If you would like, I can either give you access to the old server or transfer your database again.

Quote:

Originally Posted by lvshell702

I guess that didnt work. I just lost MySQL entries from the last couple hours.

#22 08-10-2007, 06:10 PM

I know... im trying to be patient... its the rest of my community thats stressing me out lol... I do appreciate all the hard work you guys have done to get this going again.

If I could get access to the old server so I could grab the files that were lost, that would be most awesome!

***GatorDaveC*** · #23 08-10-2007, 06:20 PM

Any accounts that are over 50,000 inodes are being transfered to the new server currently. If you are receiving 404's on your site right now, chances are that your account has used more then 50,000 inodes.

If there is anything that you would like to pull off the old server yourself, you should be able to access it via FTP at the following ip (74.53.231.210). I'll see if I can get CPanel up for convenience.

Thank you everyone for your patients, if you have any issues please submit a support ticket.

#24 08-10-2007, 06:23 PM

thank you Dave... I love you! *kisskiss*

***GatorDaveC*** · #25 08-10-2007, 06:27 PM

Quote:

Originally Posted by lvshell702

thank you Dave... I love you! *kisskiss*

No problem =)

And here CPanel and WHM for you

CPanel : http://74.53.231.210:2082
WHM : http://74.53.231.210:2086

All services on the old server should be online. We will keep it online for about 24 hours.