Hacked servers / 0 day exploit everyone read!!

***GatorBrent*** · #1 02-10-2008, 03:47 PM

A 0 day exploit has made it to the public resulting in about 1% of our servers being hacked.

Any server running kernel version 2.6.17 to 2.6.24.1. I are exploitable. This is the biggest linux 0 day exploit to come out in over a year if not years. This is not isolated to being a hostgator problem. This is a problem for any server in the world running the latest secure kernel.

All our servers are rebooting or have already been rebooted into a very old kernel that is known to be secure and not affected by this exploit. This is the only fix known at this time.

Hacked Servers last we checked a few minutes ago:

4runner, camaro, sonoma, gator364, and ram.

The hackers simply replaced the index files on all the above servers but sonoma. Which means all we need to do is change the kernal reboot and wala the server is secure again. Sonoma had a common rootkit installed on it that we can easily remove / clean / secure again. We see everything they did so it's easy to undo.

We will be able to restore your index pages from backups on all the above servers except for some customers on the Ram server. Ram was in the middle of it's weekly backup and it actually backed up some of the hacked indexes. Thus if we restored from backups it would be a hacked index. (not for everyone but some)

A few servers are going to be offline for hours while they perform fsck's. Anytime you reboot as many servers as we have a few will need a system check.

Please don't contact us via ticket, chat, or phone to ask for an update. We are completely overwhelmed and if you do reach someone all they will do is read to see if I updated this thread.

I will update you the second I have new information.

I apologize for the delay in getting this information to you. I know it doesn't seem like we were dong anything, but our admin's have been on top of the situation from the very minute this all started.

Please have patience and we will have everything fixed up very shortly.

Thanks everyone!!!!

Keith W · #2 02-10-2008, 03:52 PM

HI Brent,

Thanks for the update on the current situation.

Also thank you to everyone on the HostGator staff who are working hard to get this problem resolved.

Take care, Keith

slapshotw · #3 02-10-2008, 03:55 PM

Note to dedicated server owners:

SSH into your server and run "uname -r" to get your Linux server version. If you have a version in the affected range, I suggest contacting Hostgator to get it either downgraded or upgraded. Make it clear you're on a dedicated and need work done, rather than just asking status for the shared servers.

***GatorBrent*** · #4 02-10-2008, 03:55 PM

gator364 and 4runner should be all restored.

Ram we have restored all the indexes that we could. If your page is still hacked we won't be able to get it from backups. You will need to upload your index again.

Sonoma is being worked on now.

***GatorBrent*** · #5 02-10-2008, 03:57 PM

Dedicated servers aren't as in danger at the moment. The main reason being they usually don't have many sites on them so hackers don't know they exist. We will be figuring something out shortly for dedicated servers.

GvilleRick · #6 02-10-2008, 04:26 PM

Thanks, Brent for the information. I looked at my dedi and am running an older kernal. I wasn't too concerned though because these script kiddies usually target big hosts.

regentronique · #7 02-10-2008, 04:31 PM

Superbird is not responding anymore for the last hour, any news on that server?

***GatorDaveC*** · #8 02-10-2008, 04:32 PM

Quote:

Originally Posted by regentronique

Superbird is not responding anymore for the last hour, any news on that server?

It looks like superbird is up now, I believe this one was file system checking.

***GatorBrent*** · #9 02-10-2008, 04:35 PM

We rebooted the gator server a minute ago which is why the forums went down briefly / our site.

This is a local exploit so they would have to compromise a script or user account in order to root the box. (why most dedis are safe)

Anytime your selling hosting and have dozens of customers one of them is bound to be exploitable which is why this mostly is just going to hurt hosting companies.

regentronique · #10 02-10-2008, 04:35 PM

Nope, still no response from SuperBird for any websites neither FTP access.

Sam · #11 02-10-2008, 04:42 PM

Quote:

Originally Posted by regentronique

Nope, still no response from SuperBird for any websites neither FTP access.

Loads fine for me

GvilleRick · #12 02-10-2008, 04:42 PM

I'm seeing sites on that server. A couple timed out but I tried 15 or 20 and all but a couple came up.

regentronique · #13 02-10-2008, 04:46 PM

SuperBird just came back 4 minutes ago.

So we should expect 1 hour shutdown for a file system checking?

Was believing it would be quicker than that...

I hope all people will be back on line soon.

Thank you Brent for your prompt message.

Sam · #14 02-10-2008, 04:49 PM

Quote:

Originally Posted by regentronique

So we should expect 1 hour shutdown for a file system checking?

Yes, the larger the drives the longer the fsck will take.

venomspit · #15 02-10-2008, 04:52 PM

Thank you for the prompt attention to this. I have clients who now have this rediculous index page on their sites. I look forward to camero being back online and sunday's backup is uploaded.

I will keep checking this, and am currently drafting an email informing my clients of what happened.

***GatorBrent*** · #16 02-10-2008, 05:08 PM

The only server currently down is saturn. The dc is working on this now it is currently hanging at network configuration.

regentronique · #17 02-10-2008, 05:09 PM

Quote:

Originally Posted by Sam

Yes, the larger the drives the longer the fsck will take.

Yes i knew this, the delay is also related to the kind of HD and interface speed. I was more expecting something around 20-30 minutes for SuperBird fsck and a maximum back online delay of 45 minutes.

This is why i waited at least an hour to report the situation detected earlier...

ghpk · #18 02-10-2008, 05:13 PM

Brent sir,
i was affected with such hack earlier, Sent you a PM with info i was able to dig.
regards,

El Hombre · #19 02-10-2008, 05:29 PM

Quote:

Originally Posted by GatorBrent

gator364 and 4runner should be all restored.

Ram we have restored all the indexes that we could. If your page is still hacked we won't be able to get it from backups. You will need to upload your index again.

Sonoma is being worked on now.

So are you saying that if any sites are still showing the hacked page now, then they wont get restored?

Do you not have older back ups you could use?

Excuse my ignorance but how do I upload my index again?

I am not a designer or programmer and my sites have been developed by many different people over the years, so how and where do I get the indexes from to upload them?

I still have 3/4 of my sites showing the hacker page.

EdwardWeiss · #20 02-10-2008, 05:37 PM

Now when I go to my index page I see all my files and not the page itself. Should I FTP my index page or wait?

GvilleRick · #21 02-10-2008, 05:39 PM

If you have a full or home backup either on the server or your local computer you would just need to upload the index.html or index.php file to public_html. In a pinch I have found cached versions of pages in Google or internetarchive.com and been able to recreate the page there. It really depends on how complex the page is.

HG backs up the sites weekly but the new backup will overwrite the old. If you have an example of one of the sites you can give we can look to see what might be done.

EdwardWeiss · #22 02-10-2008, 05:44 PM

Yep. Just FTP'd up a new index file and it's working!

***GatorBrent*** · #23 02-10-2008, 05:50 PM

We have restored all that we can. If yours wasn't restored you will need to manually do this by logging into ftp going into your public_html folder and uploading a page named index.html

google cache would be good to get it from if you don't have it anymore.

***GatorBrent*** · #24 02-10-2008, 05:55 PM

Saturn has been fixed after failing to come up after reboot. All servers are on an older kernel now that is not vulnerable to this local root exploit.

I believe that concludes this emergency. We will now be working to get tickets and support back under control. If you are one of the few with messed up indexes still please put a ticket in if you don't know what to do. we can try going to google cached pages and helping.

GvilleRick · #25 02-10-2008, 06:07 PM

Isn't this one of Murphy's Laws? Seems that problems like this always pop up on weekends. Good luck to all the techs trying to wade through the tickets.

A few weeks ago I was working on a web page on our school website. I had loaded the page using notepad directly from the server. When I went to upload it using FTP there was a network glitch that I didn't notice. A few seconds later I closed my FTP program thinking the file had been uploaded but it ended up uploading a blank page. I had a recent backup at school but not on my home computer. After the initial sinking feeling of seeing a blank home page I checked out a cached version in Google and was able to recreate it in a few minutes with an older backup at home. I immediately downloaded a full backup of the site to my USB drive.