How long does it usually take?

[Solomon]

A week ago I received the following note from abuse@hostgator:

We have been forced to disable the mt-search.cgi script on the solomon account on gator177. This script was hanging causing a high server load. Please review our logs below and let us know what can be done to prevent further issues.

Hey, that's cool. My last host would disable a troublesome script without even telling me, and I'm not using mt-search, anyway (Movable Type's built-in search script). Overall, I've been very happy these couple months with HG, and things are still looking fine.

No hurry here, I let most of a week go by, contact MT support to find out if they have any advice for taming the search script (it gets churned like mad by spammers)...they get back to me, I forward their response to HG, I get back a fairly non-responsive (unhelpful) answer...I'll spare you the details. Anyway, not a big deal...I figure I'll just let it go.

In the mean-time, I install a plug-in that's supposed to auto-ban spammer IPs automatically when they start hitting comments and trackbacks, so I reactivate my trackback script to give it a try (spammers also hit this like absolute mad, so I anticipated a problem and just deactivated it myself shortly after joining up with HG).

This morning I wake up and all the scripts on my site are shut down, nor can I get into cpanel with no word from HG. Ooookay. I can guess at what happened, so I shut down my trackback script (rename it via ftp).

I grab one of those abuse@hg emails and hit reply:

9:30am

I see that all of my cgi scripts and cpanel are down. I'm assuming that that's because I re-enabled my trackback script and it started causing load issues.

I have now disabled it again, so could you enable my site again, please?

You must understand, on a blog site like mine, not being able to update anything, having no comments...it's bad...very frustrating...It's like running a news room and watching news cycles go by as people change channels.

3 hours later I hop into live support and they tell me I just have to be patient and wait for abuse to get back to me. OK, can do.

1:00pm:

We never re-enabled the web based site. We only granted your account FTP and cpanel access to fix the files. After you fixed them, you were supposed to respond back so we could re-enable the site.
Thank you.

Ooookay...my reply:

The site (scripts and cpanel) were working (other than the offending mt-search.cgi) as of last night. Only this morning was everything deactivated. Now even cpanel is not accessible.

In any case, I have "fixed" two of the questionable scripts -- search and trackback -- by deleting them. I will activate them again only when I have found a solution to the load problems they create (not likely).

Me again, at 4:30:

Again, let me emphasize, the script in question has been deleted, and one other script which could possibly be causing trouble has also been deleted...so...

So here I am at 8pm and my site is still caught in a luff, dead in the water.

My question: Am I being impatient or not assertive enough? I stated in my first email at 9:30 in the AM that I had resolved the problem (as far as I knew what the problem was). Going on 11 hours with a site deactivated seems like an awfully long time to wait.

I've had good results from HG support so far, but this seems to be an exception and it is frustrating. I understand HG's issues with making sure their servers stay up, but these replies almost sound as though they're not really...reading my replies.

[slapshotw]

Just make sure you're not replying too soon to your open tickets, or you get moved back to the queue.

I feel for you though, I do-- you seem really calm and reasonable about this whole thing which is more than I can say for what I would be like. Unfortunately we're in weekend now, and AFAIK abuse only works weekdays.

[Solomon]

UhOh, I'm gonna be a lot less calm and reasonable if my site is in this condition all weekend, especially after re-reading what I've written above.

[Solomon]

Looks like I won't need to wait...got an email 15 minutes ago from abuse who is transferring me to support. Why do I get the feeling someone flipped a switch they shouldn't have?

[Solomon]

Oh boy, 1 am and I got this (from the same person in abuse who must know no scripts are working -- no forums, no text randomizers, no comments, no blog control panel...etc...), nor can I access cpanel with my domain name):

Hello,

Where can we replicate this issue and what error messages are you receiving.

They're not just playing with me are they?

[gwyneth]

I second Matt about your attitude, which is likely to be helpful as you deal with both HG departments.

I can't provide much useful advice except that with zillions of accounts, abuse is probably working on more than just you and I'm surprised they're working on the weekend. The perception of 'fast work' is likely to be different on an affected customer's end from theirs.

Hang in there.

[Solomon]

Thanks, although it looks like my attitude isn't buying me much, but here's Sunday and still nothing to show but an abuse thread marked low priority.
It appears support is a bit more active ( http://forums.hostgator.com/showthread.php?t=15152 ) ...too bad I wasn't actually transfered there.

I'm in business, I know how it is. Screwups happen. It's all in how they get handled when the right people get ahold of the situation. I'm willing to provide plenty of leeway until that happens and see how they handle it in the end.

[slapshotw]

Originally Posted by Solomon

an abuse thread marked low priority.
It appears support is a bit more active

Don't mind the priority, they ignore those and just deal with tickets as they come in.

[Solomon]

I'm back up. No explanation (abuse: "I have verified that the banners are now loading and you are able to log into cpanel via the domain.com/cpanel."), so not much to say, but I wanted to update the thread.

[Solomon]

One thing I'll say is that Movable Type really needs some work to prevent the constant churning of scripts by spammers causing resource problems. My search, trackback and comments scripts have all caused problems at various times.

For search and comments, a way that only allows access for those coming in with the site as a referrer (no direct access to the scripts) would do it. My understanding is that mod_security (suggested by MT support) can do this but it doesn't go on a shared server.

[slapshotw]

I think they do actually have mod_security installed on the shared boxes. Check with support.

ETA: And maybe they can change the mod_security settings to help you out.

Can you provide the ticket number(s) for your issue so we may look into it?

The normal turn around time for a response to an abuse ticket (not the resolution to an abuse issue) is 8 hours. Most abuse issues can take 24-48 hours to resolve.

Basil H.
HostGator Technical Support
http://hostgator.com/help

[Solomon]

GatorBasil:

[Abuse #QQS-265578]

Thank you. It would be helpful if you could shed more light on this incident.

Slapshot, I will do that.

Me:

Thank you. Actually, I would like to avoid this situation in the future. Can you tell me if anything aside from the mt-search.cgi script was causing trouble. I only guessed that my trackback script was a problem (name: mt-renamedtb3.cgi). Can you tell me if it was?

I would like my MT installation to run on all cylinders, but don't want to risk another shut down.

Response:

When the information was captured from the server it only indicated the mt-search.cgi script.

root@gator177 [~]# ps aufx |grep solomon
root 29257 0.0 0.0 3724 660 pts/0 S+ 11:07 0:00 \_ grep solomon
solomon 254 26.7 0.7 34812 31944 ? RN 11:03 0:59 | \_ /usr/bin/perl -w mt-search.cgi

I would consider just going ahead and re-enabling the trackback script, but I really don't want to risk this happening again. It's one thing to have the particular script shut off, and quite another to lose functionality over the whole site (however that happened in this case).

The second issue where the entire site was giving an error message does not appear to be related to the original abuse issue.

The errors on the site appear to have been due to an unrelated issue with PHP on the server that was being worked on at the time the site was down.

The abuse department never disabled the website itself. They only disabled the search script that was causing a high load. This is the normal procedure for when the load issue can be isolated to a particular script.

mod_security is installed on all of our shared and reseller servers. It is configured by the system administrators to block exploit attempts. It does not provide any protection against someone using a form to send email. There would be no way for mod_security to know if the form submittal is a legitimate entry or someone attempting to use the form to send spam. mod_security is not customer configurable.

Basil H.
HostGator Technical Support
http://hostgator.com/help

[Solomon]

Thank you, Basil, that makes me feel a bit more confident that I'm not going to lose my site just because of one script issue. Who could argue with one rogue script being shut down?

The incident itself is odd, since as you describe it, I must not have been the only person with the problem, and it didn't just effect my php scripts, but cgi as well. I suppose it's all related (and pretty close to Greek to me).

FYI, here is the entire reply I got from MT support regarding the search script issue. Particularly note the mod_security part:

Hi -

The searches being submitted may be returning a large
number of results and putting extra load on the server to
compile these results. You may wish to consider using
Movable Type configuration directives such as
MaxResults[1], SearchCutoff[2], and NoOverride[3] to limit
the number of results returned by a search.

You could also try adding a SearchScript[1] directive to
the mt-config.cgi configuration file, rename the
mt-search.cgi script on the server to match the name
specified in the SearchScript directive, then rebuild your
weblog so the Movable Type search script would not be at a
known default location.

If your web hosting account is using an Apache server, and
mod_security is installed and available on your server, it
may be possible to use mod_security configuration
directives to prevent spammers from reaching the
mt-search.cgi script. You would need to contact your web
host and ask them about the availability of mod_security on
your web hosting account, as well as about what ability you
would have to configure mod_security on your web hosting
account.

Regards,

David Phillips
Technical Services

[1]
http://www.sixapart.com/movabletype/...d_configuratio
n_directives/#entry-7092
[2]
http://www.sixapart.com/movabletype/...d_configuratio
n_directives/#entry-7090
[3]
http://www.sixapart.com/movabletype/...d_configuratio
n_directives/#entry-7097
[4] http://www.sixapart.com/movabletype/...d_configuration_directives/searchscript.html

I wonder if it would be possible to place cgi scripts in a directory above public_html that isn't directly accessible, and change MT's config to point there. Or do cgi's have to be in the cgi-bin directory? I suppose if it were that simple someone would have done it already.

BTW, for those following this who use software like MT, one of the most common "spam avoidance" techniques that's recommended (and you'll see it recommended above) is to re-name your comment, TB, and search scripts from their default names. I can tell you that this is only very marginally effective. The last time I did that the new scripts began to be hit almost immediately. It's clear most of the spammers now have their bots look at the page first, then hit the new name, not to mention those that simply brute force hit every form on the page....my email-this-entry script gets spammed from time to time with nothing but gobbledy-gook -- I see the bounce message -- there isn't even any advertising information embedded in it, it's just garbage characters.

[gwyneth]

Originally Posted by Solomon

.my email-this-entry script gets spammed from time to time with nothing but gobbledy-gook -- I see the bounce message -- there isn't even any advertising information embedded in it, it's just garbage characters.

"Gobbledy-gook" is what the notorious I-frame exploits rely on (when the stream of binary numbers is interpreted it becomes a javascript command with an i-frame for another domain).

Not that this is happening, necessarily, but something to be aware of.