Virus issue has been resolved! Here is the latest news!

[GatorBrent]

After a few days of us going crazy we have solved the recent issues. Here's what we figured out...

This was a 0 day cpanel exploit. Anyone in the world running cpanel could have been exploited.

They actually did the cpanel exploit about a month ago which explains what we thought at the time to be "bad cpanel updates." We thought this because sites weren't loading in IE and the fix was just changing a line in cpanel. At this point in time viruses weren't loading as far as we knew or heard so there was nothing to suggest anything different then a bad cpanel update.

We believe whoever did this was perfecting what they were about to launch and waiting for the right moment. They chose a few days ago to launch it in full force to exploit Microsoft's newly announced vml exploit. They used the exploit in cpanel to distribute trojans / viruses to target the vml exploit.

Here's what cpanel said once we showed them the exploit....

Originally Posted by Cpanel

"This has been confirmed and patched. Running /scripts/upcp will fix the vulnerability in all builds."

We had our own patch we ran before the release of cpanel's, and as soon cpanel provided an update we ran their patch as well.

We had a few problems to figure out......

1. How was it happening and where was it coming from?

We could easily fix the problem but every time we did in minutes to an hour later it would come back. After hours of looking how this was happening we made little / no progress. We reached out to the web hosting community for help and soon had everyone helping us.

To name a few....

ThePlanet's security team, Layeredtech's security team, idefense.com, verisign, our best inhouse administrators and gurus, some server admin companies, and a customer of our's named Brad who helped build the architecture of paypal and ebay.com.

Brad had some contacts in symantec, trend, and Mcafee that he was able to contact on our behalf. We had everybody working on this. Our CTO DaveC finally figured out what was causing it and cleaned it up at which point it has not happened since.

2. What was exploited and how?

We might have cleaned it up to fix the problem, but without knowing how they were exploiting our boxes they could easily do it again and again. One of our best admins Tim Greer solved this mystery today when he came across a cpanel root exploit that nobody knew about. He tested it and it was soon proven this cpanel root exploit is how the hackers had the power to do the redirects. As soon as we knew the function of cpanel that was being exploited we had help with the creation of a bandaid patch that was applied immediately.

At the same time this was going on I got on the phone calling everybody in the industry to get cpanel involved. I was able to reach cpanel's operations manager Dave who quickly came up with a patch that has now been released to the public. At this point we ran upcp which will prevent our boxes from being exploited this way again.

3. Where are we at now?

A lot of people that use Internet Explorer got viruses and will need to run a virus scan. I'd appreciate if some affected people could post the best way for scanning and removal. We do not have any evidence of anyone's passwords or personal information being taken, but to be safe it would not hurt for everyone to update their passwords to something complicated.

Hostgator's boxes have all been cleaned and the cpanel patch has been applied to avoid this from happening again. All other hosting companies that haven't applied this patch are going to get it installed automatically tonight. Many of them will remain exploited until they clean their boxes as we did.

The person or group that did this is very intelligent, and obviously knows how to plan a big attack. While we are protected from this threat we cannot predict what's to come for hostgator and the industry. Nobody can. No server is 100% secure. There could be a new 0 day exploit around the corner that takes out the entire internet. Anything is possible. We will continue to stay on top of security and do our best to provide the best possible hosting experience.

I realize our staff gave a lot of people wrong information. The truth is we weren't really sure what the problem was for a while, and we were actively working on it. There was a lot of trial and error involved. Had we simply just turned everyone off to avoid prorogation of the virus we would not have been able to peform the trial and error needed to trace it down to cpanel being exploited.

Our staff was responding to tickets saying they couldn't reproduce some of the issues, this was likely because we would fix it and respond back to the ticket at which point the time you read it the problem was back again.

4. Why were we targeted and by who?

We were most likely targeted due to our size and solid reputation. Since this exploit could have worked on anyone running cpanel it had nothing to do with how secure we were. We suspect it was done by someone in china as we have a small piece of information supporting this. I wish I could provide more specific information related to the exploit, but the less people know about it the less likely it will popup in a different variation.

We apologize to everyone for this issue, and if there was anything we could have done to avoid it or solve it quicker we would have done it. I would like to thank everyone that helped us in this crisis. I'm sure the situation could have been handled better, however we did the best that we would with the man power we had available to take calls, chats, and tickets around the clock. A tech usually takes 4 chats at a time while this was going on many had 20+

Thank you for understanding. This was devasating to us as well as anyone that had a website affected. We will do our best to help everyone recover from this.

[MattFriend]

Hi Brent:
Thanks for the explanation. Just so I understand better... the fact that my server (158) is currently down (and has been off and on most of the day) is not related to this attack? Matthew

[newhall]

Thanks Brent! I really appreciate your personal involvement and assistance in getting everyone to pitch in and get to the bottom of this horrible exploit and resolve this emergency. Your hand's on leadership and support is very much appreciated. Kudos to you, your staff, and all those that volunteered to help HG and us out. THANK YOU!

[GatorBrent]

MattFriend, your issue has been fixed. It was a user on your server gettng insane traffic maxing out the apache sessions. I have suspended him, and I'm guessing it took so long to fix since were still so overloaded from everything else that happened. (Probably haven't gotten to anyone notifying us yet till I read your post)

[TakeThat!]

Wow, that is quite an amazing story!
It's a shame that there are so many exploits for IE floating around out there.

Although I am a very recent Hostgator customer who's site wasn't affected, I am very glad to see that you guys were able to solve this problem!

From reading this forum I could see how badly this exploit was messing up peoples sites. Really a nightmare!

Congrats to everyone who reported the problems that their sites were having and also to all the people who fixed this thing!



While I didn't get infected from this exploit, I would recommend Ewido anti-malware to those of you that did. Also, if you don't have an anti-virus, try AVG as well. Both are free. I'm not a computer security expert but I have had alot of luck with those 2 programs.

http://free.grisoft.com/doc/5390/lng/us/tpl/v5 <-- you can get them here.

Note: You might have to run a scan from safe mode to get rid of the junk you got. You might also have to disable system restore before you scan. Malware always hides in there as I'm sure alot of you know.

The best of luck to all of you in getting your pc's clean and getting your sites back on track.

[squirrelproductions]

So if one of my sites was acting funky the other night (throwing up what looked like binary or just header information), does that mean the HG server that my site is on was exploited? I thought the funky output stuff was just a problem of one of my site's scripts because another domain on the same server was seemingly unaffected, but could it have been the buffer overflow flaw?

Is there a list of servers that were known to have been infected/exploited?

And, for cleanup purposes, what AV/malware-detection software have the virii/trojans in their dbs now? I see no sense in running a scan and getting a false negative, if the AV/malware-detection software updates don't detect the virii/trojan in this exploit.

Thanks.

Thanks Brent. It's good to know what was going on.

I understand the need to keep the system up to test out hypotheses. However, why didn't you shut off PHP? It seems that you could test the CPanel exploit without have PHP on. This would have prevented quit a few infections of users.

I like to think that I am capable enough to clean my own computer, given the proper instructions. However, I have a small and technologically naive group of users that will need instructions on how to remove any of the bad stuff.

I've also noticed that just about everyone seems to have different issues. Some of my users had pop ups, but I had none. I was infected none the less. Here is my experience so far, in case it will help others.

After following a couple leads from this forum, I found a few suspicious files. 2 I deleted with no problem, "mlljk.exe" and "mlljkl.dll", which seem to be associated with the Vundo Trojan. 2 others I could not seem to delete (even in safe mode). These are "lmhlln.dll" and "jkkljgh.dll", both are in my WINDOWS>>System32 folder

Symantec Antivirus (updated about an hour ago) seems to have tagged "jkkljgh.dll" as a trojan, but I'm still unsure about "lmhlln.dll". The former has been deleted, but the latter is still firmly in place (can't delete it).

As for my users, doing a System Restore back a couple days seems to have worked well. Never know what nasties are still hanging around, but no pop ups for now.

I intend on sending them instructions on how to clear their cache and run and updated virus scan. Not sure what else I can do now.

Any further help would be appreciated.

Sorry, the file name that I cannot remove is lmh11n.dll, not lmhlln.dll (get those 1's and l's mixed up)

If anyone has a copy of the virus that was distributed feel free to forward it to me so I can run it through a few different scanners on a test box here at home.

If you don't happen to have a virus scanner yet and one of the reasons being money there is a link earlier in this thread for AVG and my personal recommendation goes out for:

AntiVir: http://www.free-av.com/ -- It's free, great and updated very frequently.

If you need to generate new passwords there is a link in my signature to a script that will generate a few to use.

Just a sidenote:

In regards to us (the techs.) having trouble reproducing the issues: Both IE7 and Firefox were unable to reproduce the virus redirects (or were automatically blocking them) which lead to some initial confusion (at least on my part) when resolving tickets.

[MattFriend]

Originally Posted by GatorBrent

MattFriend, your issue has been fixed. It was a user on your server gettng insane traffic maxing out the apache sessions. I have suspended him, and I'm guessing it took so long to fix since were still so overloaded from everything else that happened. (Probably haven't gotten to anyone notifying us yet till I read your post)

Thanks for responding Brent and fixing the problem. I have a question though (I guess my linux knowledge is limited): wouldn't someone maxing out hte apache sessions cause the cpus or memory usage to spike? If you check in the other thread on server 158 I inserted a screen pic and it seems to show low memory/cpu usage...? Matthew

Originally Posted by MattFriend

Thanks for responding Brent and fixing the problem. I have a question though (I guess my linux knowledge is limited): wouldn't someone maxing out hte apache sessions cause the cpus or memory usage to spike? If you check in the other thread on server 158 I inserted a screen pic and it seems to show low memory/cpu usage...? Matthew

Matthew,

Not necessarily. The limit is actually in place to prevent the cpu from spiking too high. As far as memory goes: The way *nix works is it basically doesn't allow any free memory to sit around so it's almost always constant (and full).

Resolved?? Got news for you! I thought I got rid of my virus but there were two more instances on my computer again. No sooner did I delete them when a new one showed up. I deleted them and I got a pop up from Norton that I have another virus. This one is a totally different one. Are the sites still sending this out?? If so I need to remove mine again.

Kelly

[L146705]

thanks for your information and work on resolving the issue. Your customer service was awful during the crisis, you failed to give much information on what was happening and you also gave wrong/bad information such as refresh the webpage. You will need to look into your customer service for this crisis and address the issues so that it doesn’t happen in the same way again.

Also you should have taken the sites down rather than let them give viruses out.
Do you honour your 99.9% moneyback gaurantee?

I spent a day for scanning my PC, because during those hours before I knew my site was hacked, I was using my credit card, paypal account, etc, and I don't know what info has been stealed, and I am so no-headed what should I do now because I think there are too much things to do... I can't remember which of my passwords had been inputed during that period and how to go through the complicated procedure of cancelling my CC and change my paypal's password.

When I saw on the news that these guys may come from china, I was so sorry. Because I'm chinese, and can't believe these un-educated guys could do such a immoral thing. It's such a shame to the hosting industry, and can not be forgived.

I finally appreciate HG's hard works on this, and really hope a peaceful hosting future to everyone of us here. However, for these guys who did this, I highly suggest HG can do something to punish them, like to call an international litigation, etc.

Best regards.

TT

hi,
Those who are infected should not Take chances in just running a scanner, Even prominent AV softwares like mcafee and norton still don't have detected infection on many systems.

But i was PRETTY sure that iam infected with the code from the I.E. exploit.
cause i never visit PORN sites, i got pop up's with it.

I didn't bookmarked some antivirus selling site, its there and opening.
even when norton says it has cleaned the 2 viruses i get the popup every few hours, and when i check the running processess i See _wowexec a process which i doubt replicates itself every few minutes in a new Name.

so morale of the story is, Even microsoft doesn't know how many viruses have been built with this exploit, so why should one wait for a disaster to happen to your private information on your computer " FORMAT YOUR SYSTEM"

If this was a Bug, it was with us all the time after Service pack 2 release, so why the hell i trust the current installation of windows xp, infact stupid's XP.

Anyone who is serious about his mails, passwords, billing infos residing on system, just format the damn drive and re-install everything.


Hope to have more anti-virus suggestions.

[Serra]

I'm currently using Kaspersky http://www.kaspersky.com for virus scanning. It is very solid, never crashes my machine and I just got a free upgrade from version 5.0 to 6.0. This scanner is very technical, so if your a technical person, you'll love it. (Those who aren't should stick with plug and play scanners).

Brent, thank you for all your work.

Everyone else, if you still think that Hostgator is to blame for this problem, please go to the cPanel forum and read the posts there. Brent gave us far more information than they have and it was THEIR problem. I'm still very happy with Hostgator and its too bad they where attacked, but they reacted properly and solved the issue. The support people don't always give us the correct information and as Brent said, some times they gave us information that was totally wrong, but in a crisis, they did the best they could do. They aren't perfect, neither are we!

[L146705]

try avg free and
spybot search and destroy(its free) - on the download page u may need to scroll a bit its below the donation button.
http://www.safer-networking.org/en/download/index.html

Originally Posted by Serra

I'm currently using Kaspersky http://www.kaspersky.com for virus scanning. It is very solid, never crashes my machine and I just got a free upgrade from version 5.0 to 6.0. This scanner is very technical, so if your a technical person, you'll love it. (Those who aren't should stick with plug and play scanners).
...

Just for people to know, that I was infected yesterday when I was with Kaspersky 6.0. Ewido founds the viruses.

[Kostas]

Thank you very much for your great support.

Brent, you must be proud for your support team...!

[Linkin]

*EDIT* My issue must have been a reboot. I'm back up.

Thanks.

First off, right now money is extremely tight.

Please help:
Has anyone used Kaspersky with Win2000/sp4? and XP/sp2?

I also want to remove my AVG free version on both systems and install the 30day trail of Kaspersky.... does anyone see any problems there?

Is Kaspersky the best for spyware/malware/keylogger detection?

I've already scanned 3 times with AVG(latest update), SpyBot SD(latest update), AdAware(latest update), Old 6.0 version of SpyCop, I also have TeaTimer, a realtime registry change monitor, running. I do not and did not get any popups. I did access infected redirect pages using both my machines and IE. The scans did produce a large amount of crap on both machines.

I am very worried about entering my personal data on a daily basis, nec. for my bus.

Ideas PLEASE,
I have to secure my systems fast so that I can get back to producing my family’s only source of income. Been down 48+ hours.
Mark

PS Yes!, I no longer use IE, even though FF is slow loading for the type of work I do.

[tcolling]

Is there anything that WE (HG's customers) need to do to make sure that the server-side problems are solved, such as "updating" cPanel, or will HG make that happen automagically for all of us?

- Tim

[Kelmas]

Now going back to the virus. I opened the malware-framed site andprobably got the virus. I was logged in as limited user, so I think OS files were not harmed. I saw test[1].exe file in ZoneAlarm program list, but got it deleted with cache cleanup.

I have scanned my system with the following:

• ZoneAlarm Security Suite (latest)
• NOD32 (not latest engine, but current virus definitions)
• AVG Free (latest)
• Spybot Search & Destroy (latest definitions)
• ewido anti-spyware (latest)

ewido found some malware - Blackdoor.Pcclient and deleted it. And that's it. No other threats were discovered by mentioned programs.

Any more responses about that virus?

[97transam]

Originally Posted by Serra

...
Brent, thank you for all your work.

Everyone else, if you still think that Hostgator is to blame for this problem, please go to the cPanel forum and read the posts there. Brent gave us far more information than they have and it was THEIR problem. I'm still very happy with Hostgator and its too bad they where attacked, but they reacted properly and solved the issue. The support people don't always give us the correct information and as Brent said, some times they gave us information that was totally wrong, but in a crisis, they did the best they could do. They aren't perfect, neither are we!

Agreed. I think we are all dang fortunate that Brent and HG has connections to some top qualified personnel to work on this situation on our behalf.

Anyone else see that there's a javascript error on this HG forum thread or is it just me?
--ron

So, if my site was infected on one of your servers, is there anything I need to do on my site to clean it, or have you taken care of what ever was causing it?