Emergency Maintenance Lebaron

[GatorJoshL]

Lebaron has a failing drive, our datacenter is going to be taking the server offline for an emergency swap. I will keep you updated on the situation.


Datacenter is taking this server offline right now.

[GatorJoshL]

The server is up and rebuilding the RAID Array at this time.

[keichi.mail]

Hi, nothing to say about maintenance, but the ftp service is down days ago, saturday, sunday, monday and so on.... email ok, web ok, but ftp is totally down, i have been calling from colombia 3 times saturday and many support chat, ftp is still in the same status

Thanks

[keichi.mail]

now is not working the email....again... the ftp is gone too, specting calls of my customers, i dont not how explain them, 3 days on go on, guys...please

[GvilleRick]

Originally Posted by keichi.mail

now is not working the email....again... the ftp is gone too, specting calls of my customers, i dont not how explain them, 3 days on go on, guys...please

If you have not done so you should open a ticket with support so that the techs can have a look. Once you open a ticket you might contact Live Chat to ask them to have an admin have a look.

This forum is primarily peer support and techs do not regularly check the board.

[keichi.mail]

i did 3 calls from colombia, also i have opened 4 tickets or above, live chat all the time to know all new about it, i do my best too...

[GatorJoshL]

Hello,

At this time the rebuild of the raid array is at 90% Complete, this causes very high disk IO and might cause certain services to fail on and off till the raid is completely rebuilt. We do apologize for any inconvenience, if you have some tickets please PM them to me and I will be glad to assist you with them.

[verdecove]

Originally Posted by GatorJoshL

Hello,

At this time the rebuild of the raid array is at 90% Complete, this causes very high disk IO and might cause certain services to fail on and off till the raid is completely rebuilt. We do apologize for any inconvenience, if you have some tickets please PM them to me and I will be glad to assist you with them.

Having problem with my site is loading with totally different speed from two not too far from each other locations and similar internet connections.
Using AT&T DSL in Spring TX (2mb download speed) site loads very very slow and using Comcast Cable here in Houston site is loading fast. Is this caused by any issues with the data center?
I noticed the traffic of my site has gone down drastically since last Sunday and a customer of my store has emailed me and told me the site was loading extremely slow.
Please advice.
Thank you
verdecove
http://cleanstar-homeandgardenworld.com

[bobchmara]

On Sunday, about 24 hours after the new drive was installed, the rebuild was at 72%. Yesterday, after 48 hours it was at 82%. Today, after 72 hours, it's at 90%. This suggests to me that something else is wrong. Further, I periodically got email on Sunday and Monday, but so far, I've received no email today, nor can I access any mail accounts via webmail.

I don't like to complain, but suggesting that someone open a ticket is meaningless. This isn't some anomaly that can be repaired. The rebuild has to run it's course. I'm questioning why it is taking so long for that to happen.

[GatorJoshL]

Hello,

The drive is at 91% Rebuild:
//lebaron> info c0

Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC
------------------------------------------------------------------------------
u0 RAID-10 REBUILDING 91 64K 931.303 OFF OFF OFF

Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 OK u0 465.76 GB 976773168 9QG3YGF8
p1 DEGRADED u0 465.76 GB 976773168 9QG3SV0H
p2 OK u0 465.76 GB 976773168 5QG0CD5T
p3 OK u0 465.76 GB 976773168 5QG0CDBV

Name OnlineState BBUReady Status Volt Temp Hours LastCapTest
---------------------------------------------------------------------------
bbu On Yes OK OK OK 0 xx-xxx-xxxx

//lebaron>


Rebuilding these Drives takes a little while because of the other activity on the box.

We are making a few changes on the server right now to hopefully speed this process up. We are aware of the DNS issues and mail issues at this time and are working to get this drive finished repairing so that the server is back online and functioning as it was.

[GatorJacob]

Hi Bob,

The main reason it's taking so long is that the server is really really busy. Since it's so busy, we have to drop the rebuild rate to 1 (this is the lowest rebuild rate). Dropping this to 1 allows faster I/O for other processes, such as email, Apache, MySQL, etc etc. Dropping it to 1 also make it rebuild really slow.

You can imagine if we had the rebuild rate set to 5 (highest 3ware servers can go), nothing on the server would work period.

I apologize for the inconveniences and the frustrations that this has caused for everyone. Please note that we are doing everything we possibly can to keep this server up to speed, however our hands are kind of tied with what we can do, what we can put in RAM, etc etc.

The rebuild is currently at 91% complete. Hopefully this will finish today so services will return to normal.

[keichi.mail]

Could any admin please take care of the email service?? One day without email service????

[tr2112]

Thanks guys. Please keep us posted on the rebuild. I have clients calling non-stop regarding email service. I appreciate all your help.

[keichi.mail]

sorry for fill up this again but...now the entire server is offline...email, fto, web, all stuff are crashed!! gee... :-(

[GatorJacob]

Hi,

The server is up and running at this time. Trust us, if the server or services are going down, we're the ones doing it. We're trying to make changes and updates to the services to help them respond better. It seems to be helping as the load is lower.

We'll be monitoring this server very closely.

[GatorJacob]

We've paused the rebuild at the moment. We just couldn't hold it stable enough with the rebuild going and services going at the same time.

The rebuild will continue later this afternoon / evening.

[memoc@n]

Now I want to die

[memoc@n]

I thank God all accounts currently running email and http server is running. I hope I will not go again ...

[GatorJustinM]

We have resumed the rebuild on this server. We are monitoring the services very closely. Please note that the rebuild completion is at 91% and the sooner this is done, the less issues will see thereafter. We apologize for the continued troubles on this server but we would rather have this completed asap than to risk any kind of data loss which could make the situation worse.

[mahinda]

Till you done the rebuild work, cant you move our account to another server? Because we are in deep trouble with our client complaints.
What is the estimated time to back it normal I mean 100%?

[GatorSLadewig]

We are continuing to monitor the server very closely. Currently the raid rebuild is at 98%, and we anticipate that it will be completed shortly. We apologize about the continued inconvenience of this.

[EarlyOut]

Originally Posted by mahinda

Till you done the rebuild work, cant you move our account to another server?

The problem is that your files are on the drive that's being rebuilt. How could they move your files to another server?

[Abomination]

Originally Posted by GatorJoshL

Hello,

The drive is at 91% Rebuild:
//lebaron> info c0

Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC
------------------------------------------------------------------------------
u0 RAID-10 REBUILDING 91 64K 931.303 OFF OFF OFF

Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 OK u0 465.76 GB 976773168 9QG3YGF8
p1 DEGRADED u0 465.76 GB 976773168 9QG3SV0H
p2 OK u0 465.76 GB 976773168 5QG0CD5T
p3 OK u0 465.76 GB 976773168 5QG0CDBV

Name OnlineState BBUReady Status Volt Temp Hours LastCapTest
---------------------------------------------------------------------------
bbu On Yes OK OK OK 0 xx-xxx-xxxx

//lebaron>


Rebuilding these Drives takes a little while because of the other activity on the box.

We are making a few changes on the server right now to hopefully speed this process up. We are aware of the DNS issues and mail issues at this time and are working to get this drive finished repairing so that the server is back online and functioning as it was.

Where can I learn more about these types of things?

I would like to find out how these things normally work, e.g.:

1. Does a complete copy of the drive contents exist on p0, p2, p3 without p1
2. Could that contents be copied.
3. Why in these types of cases, isn't the server shut down to expedite the rebuild, and if it were shut down how long would that take?
4. How old are the recent backups.

Things like that.

It appears that this issue started Saturday morning, and it is now Wed.

[EarlyOut]

Originally Posted by Abomination

Where can I learn more about these types of things?

http://en.wikipedia.org/wiki/RAID

[Abomination]

Originally Posted by EarlyOut

http://en.wikipedia.org/wiki/RAID

So a complete copy of the disks contents currently exists, and is readable?

RAID 10

RAID 1+0 (or 10) is a mirrored data set (RAID 1) which is then striped (RAID 0), hence the "1+0" name. A RAID 1+0 array requires a minimum of four drives: two mirrored drives to hold half of the striped data, plus another two mirrored for the other half of the data.