4/6/2009 Networking Issues

[GatorLBrower]

Our data center is currently experiencing networking issues and is investigating the cause. We will post further information when it is available.

[virtvir5]

Well, I don't mean to be too inquisitive, but it would be nice if "the center" could be persuaded to be just a little bit more forthcoming this time. Non-explanations like "networking issues" and "routing issues" tend to wear a little thin after several repetitions.

[GatorEricS]

Hello,
We have spoken with our data center technicians, who have informed us that the NOC is working on the routing issue, and is aware of the cause. Although we do not yet have a time line when this will be fully resolved, we are seeing dramatic improvements in the networking issues we were experiencing.

We will continue to work with the data center and the DC NOC OPS to resolve this situation and bring the HostGator network back to full functionality as soon as possible.

[virtvir5]

I understand that you are doing your best to resolve the "networking issues" with the NOC and I'm certainly not trying to make a mountain out of a molehill. Nevertheless, I sure hope that the NOC understands and is explaining the problems and their root causes more fully to you than we are getting here. This is now the third such "issue" since this time last month and the non-explanations are not great confidence builders as this end user perceives the overall situation.

[kekuhn]

Any word on when this issue will be resolved?

[GatorBrent]

I'm not sure when exactly this was resolved but it should have been resolved a long time ago. I'm about to hop on a conference call with their CEO for an explanation. Last I heard, it was their theplanet.com name servers that were under a major DDOS.

[GatorBrent]

I just got off the phone meeting with them. Here's is the latest official response....

"Between 10:15pm CDT on April 6 and 1:15am CDT on April 7, The Planet's
authoritative name servers (ns1.theplanet.com, ns2.theplanet.com) were
flooded with a very large and brute force Distributed Denial of Service
(DDoS) attack. Given the volume of the attack, our network operations
team rerouted all traffic to our name servers through our DDoS
mitigation capabilities so we could isolate and clean the attack traffic
. This mitigation permitted us to resume normal operations by routing
legitimate DNS traffic through our name servers. The Planet's network
and DNS performance have been restored and the attack originator has
ceased actions. Any lingering issues may be indicative of a different
problem that may have been exacerbated by the attack and should be
resolved quickly."

There still seems to be a few issues lingering. If anyone is having a problem still related to this, please post your ticket number so that I can work with The Planet's guy in charge of networking.

[b4padmin]

Originally Posted by GatorBrent

I just got off the phone meeting with them. Here's is the latest official response....

"Between 10:15pm CDT on April 6 and 1:15am CDT on April 7, The Planet's
authoritative name servers (ns1.theplanet.com, ns2.theplanet.com) were
flooded with a very large and brute force Distributed Denial of Service
(DDoS) attack. Given the volume of the attack, our network operations
team rerouted all traffic to our name servers through our DDoS
mitigation capabilities so we could isolate and clean the attack traffic
. This mitigation permitted us to resume normal operations by routing
legitimate DNS traffic through our name servers. The Planet's network
and DNS performance have been restored and the attack originator has
ceased actions. Any lingering issues may be indicative of a different
problem that may have been exacerbated by the attack and should be
resolved quickly."

There still seems to be a few issues lingering. If anyone is having a problem still related to this, please post your ticket number so that I can work with The Planet's guy in charge of networking.

I've just submitted a support ticket for the slow load times on my site. The ticket number is:
PCG-4297581

[virtvir5]

Originally Posted by GatorBrent

I just got off the phone meeting with them. Here's is the latest official response....

Thanks for that response, Brent. I suspect that I'm probably not alone in wanting a little more explanatory info than merely "it's a networking issue" when these NOC glitches happen. And I suspect also that their knowing the top management of HG is concerned does no harm.

So their primary nameservers are getting whacked with DDoS attacks. I thought I recalled some earlier announcements here about hardware upgrades that were supposed to ameliorate that problem. Was that only for HG's own switches? Are you satisfied that The Planet's own countermeasures are adequate for the longer term?

[plumwd]

My sites are all down with a cryptic "Under Maintenance
We are currently performing scheduled maintenance. We will be back soon! " which is not my message. Is this related? I have a ticket open HAM-4298712

[GatorLBrower]

This is because the server is being upgraded to MySQL version 5.1. Please reference the following forum post for more information.

http://forums.hostgator.com/mysql-up...00-t48288.html