Phplist Throttling :: How To ::

[GatorShashank]

Hello Folks,

I hope all of you must be aware of Hostgator's Mail policy. The policy urges all you phplist/mailing lists users to throttle your mailing lists to keep them under the 500 emails/hour limit.

I am presenting specific instructions on how you can tweak PHPList available through fantastico to obey the limit while sending mails preventing legitimate domains like "yahoo.com" from being bouncing back as "unroutable domains"

Here we go :

1. Locate the config file for your phplist installation.

If you have installed the script on your domain with URL like http://www.domain.com/list/ then the physical path will be :

/home/username/public_html/list/config/config.php

In FTP you could see

/public_html/list/config/config.php

2. Download this file to your local PC.

3. Edit the file in a text editor like notepad or wordpad.

4. Locate the section called as "batch processing".

5. Just below you will see some code like :

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled
define("MAILQUEUE_BATCH_SIZE",0);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",3600);

These are the default settings. We will be replacing those to keep the mails under 500/hour.

Here is the code I suggest :

# define the amount of emails you want to send per period. If 0, batch processing
# is disabled
define("MAILQUEUE_BATCH_SIZE",480);

# define the length of one batch processing period, in seconds (3600 is an hour)
define("MAILQUEUE_BATCH_PERIOD",3600);

These settings tell phplist to send 480 mails per hour. The remaining 20 emails per domain can be used for normal mailing operations if needed.

6. Save the file and ftp it back to its original location. Remember to take a backup of the original file just in case you mess up.

Now you should be able to send mails without any bounces and not overloading the mailserver.

Any additions to this article are always welcome :-)

[amartins]

Cool, so the 200 email per hour limit _does_ apply to the mailing lists as well!

I had assumed, up till now, that the limit did not apply to mailing lists as long as you kept them throttled as per HG's specs.

Antonio.

[bkoconnor]

I am doing this and I'm having some problems with it.

When I process the queue it sends the first 175 messages, but then the browser times out in the hour down time. When I log back in, it won't process the rest of the queue, but instead gives me this message:

Started
Sending in batches of 175 emails
Processing has started, 1 message(s) to process.
It is safe to click your stop button now, report will be sent by email to ben@hibeams.com
Processing message 6
Looking for users
Found them: 202 to process
Script stage: 5
Finished, Nothing to do
0 messages sent and 175 messages skipped
Finished this run
Finished, All done

When I click the messages tab, I get:

Status = inprocess and Action = 175 done.

Any ideas on what's going on?

[mw5cents]

bkoconnor,

did you ever figure out what to do? I have PHPList installed for a client, they have not reached 200 users yet but it's only a matter of time.

[Serra]

Originally Posted by amartins

Cool, so the 200 email per hour limit _does_ apply to the mailing lists as well!
I had assumed, up till now, that the limit did not apply to mailing lists as long as you kept them throttled as per HG's specs.

That is important becuase exim is likely set to "Silent Drop" all emails that exceed the limit. In other words, send as many as you like, only the first 200 go! The setting is fairly strict:

The maximum each domain can send out per hour (0 is unlimited): XXX

Anything beyond the limit is dropped.

[monyamic]

Wait a minute. Is the limit a batch of 200 AT ONCE or 1 every 6 secs (more than 200 per hour) if sent one at a time, or either?

[monyamic]

Originally Posted by Serra

That is important becuase exim is likely set to "Silent Drop" all emails that exceed the limit. In other words, send as many as you like, only the first 200 go! The setting is fairly strict:


Anything beyond the limit is dropped.

Oh my golly gosh! that means all the newsletters I have been faithfully sending (and watching phplist send , so I thought) haven't been getting through.

[Hoggs74]

Originally Posted by Serra

That is important becuase exim is likely set to "Silent Drop" all emails that exceed the limit. In other words, send as many as you like, only the first 200 go! The setting is fairly strict:


Anything beyond the limit is dropped.

I really need some more explaination on this. Below is a copy of my trouble ticket with HG support. HG's answers seems contradictory to me. What I really want to know is if my list is over 200 members, will the remaining members get their mailing in the second hour, or will they just be lost in the mix?

dhm Posted On: 13 Jul 2006 09:40 AM

Support,


Hello, I am a reseller with Host Gator ( www.pulpwebsolutions.com ). I have a client that is a church. I have created their website (www.northjacksonchurchofchrist.com ). They have asked me to get things setup for a mailing list. They want members of the congregation to be able to subscribe to their weekly bulletin (http://www.northjacksonchurchofchrist.com/subscribe.htm). The church has over 600 members at this time. The weekly bulletin will be sent out as either an html e-mail or as a PDF (yet to be determined). Anyway, my questions are concerning the e-mail policy of host gator, and how is the best way to send out the bulletin to potentially 600+ members, and probably others that are not members.



Currently I have installed PHP List using fantastico and I believe to have
set up the throttling to 200 e-mails per hour at a rate of 1 every 6
seconds. I did this in the config file. My concern of this however is after
reading several posts on the message board is that any number beyond 200
will be dropped and never sent. I have also installed mailman using cPanel
under the e-mail section. Using mailman, I created a mailing listed titled
bulletin@northjacksonchurchofchrist.com . My concern for this program is
basically the same, plus it has not throttle capability.



So, as a reseller, I am asking Host Gator to please inform me of the
appropriate manner to go about sending out around 600 e-mailed church
bulletins once per week. I am sorry to be sending this to support, but
after reading the forums over and over; I seemed to be more confused than enlightened.



Here is all of my account info:



Reseller Account: pulpwebsolutions.com



WHM User: ********

WHM Pass: ********



cPanel User: *******

cPanel Pass: *******







Domain to use mailing list: northjacksonchurchofchrist.com



cPanel User: ******

cPanel Pass: *********





PHP List User name: njcoc (http://www.northjacksonchurchofchris...php_list/admin )

PHP List Pass: **********





Sincerely,





David Moore

Pulp Web Solutions

dhm@pulpwebsolutions.com

731-217-4623












Alex Posted On: 13 Jul 2006 10:18 AM

Hi,


Phplist would be your best bet. I personally would send alittle below 200 to be on the safe side seeing as 200 is for the whole domain for the hour. And yes if it goes over 200 the emails will bounce back to you and never be sent.



Best Regards,
Alex
HostGator Technical Support






dhm Posted On: 13 Jul 2006 10:29 AM

Support,

Thanks for you reply, but how do I go about sending more than 200. Say for instance, I am sending out a bulletin to 600 members. PHP List sends it to members 1 thru 200, and members 201 thru 600 are just out of luck or what.

Can you please explain in more detail how I go about sending a bulletin to
600 and not to just 200 people? Your rules clearly make reference to sending out mailing lists of over 900 is only allowed in off peak hours.

This is a quote form Host Gator web site.

"Mailing Lists Rules:

1. Anytime you're sending a message, no matter how large your e-mail list is you must throttle it. We recommend you throttle it to at the very least
sending 1 email every 6 seconds. If the mailing list software you're using
does not allow you to throttle, you must switch to something else. We
recommend phpList, which can be found in your control panel under
Fantastico. If you do not throttle and you try sending, for example, 500
emails, the server will try sending all 500 in 1 second, which is not
possible. This will cause the server load to go very high and the entire
server will be sluggish until this process is completed. It is our job to
keep the server up and running without being sluggish. Anyone who causes the server's load to go high will be suspended and the process will be
terminated. If you choose not to throttle, you will most likely be suspended
for crashing the server.


2. Any mailing list over 900 emails is only allowed to be sent to during
off-peak times to prevent high server loads. Off peak times qualify as all
day Saturday and Sunday, and 1 AM - 8 AM Eastern Time on Monday - Friday."


Sincerely,


David Moore
Pulp Web Solutions
dhm@pulpwebsolutions.com
731-217-4623


Alex Posted On: 13 Jul 2006 11:11 AM

Hi,


You will need to use the throttle function with phplist to do so. This will send to all 600 just during the corse of a longer peroid of time. You will need a dedicated server to send anymore then 200 per hour. 900+ email lists will need to be done sometime during non peek hours such as 1am - 6am.



Best Regards,
Alex
HostGator Technical Support

[Hoggs74]

The reason I say all this sounds contradictory from HG support is, in his 1st reply he says that:

Phplist would be your best bet. I personally would send alittle below 200 to be on the safe side seeing as 200 is for the whole domain for the hour. And yes if it goes over 200 the emails will bounce back to you and never be sent.

Then in his 2nd reply he says:

You will need to use the throttle function with phplist to do so. This will send to all 600 just during the corse of a longer peroid of time. You will need a dedicated server to send anymore then 200 per hour. 900+ email lists will need to be done sometime during non peek hours such as 1am - 6am.

[Hoggs74]

I sent a reply to HG support and it reads below:

dhm

Posted On: 13 Jul 2006 01:25 PM Support,

Sorry to keep bothering you, but your answer seems a bit contradictory. In your 1st reply you told me: " Phplist would be your best bet. I personally would send alittle below 200 to be on the safe side seeing as 200 is for the whole domain for the hour. And yes if it goes over 200 the emails will bounce back to you and NEVER be sent."

Yet in your 2nd reply you told me: " You will need to use the throttle function with phplist to do so. This will send to all 600 just during the corse of a longer peroid of time. You will need a dedicated server to send anymore then 200 per hour. 900+ email lists will need to be done sometime
during non peek hours such as 1am - 6am."

So I am still at a loss. Will PHP List either ?

A. Send the remaining e-mails over 200 in the second & third hours, or
B. Will it simply skip all the e-mails from 201 and up.

If PHP List will send the remaining e-mails in the hours to come, then is there any settings besides the 200 per hour and 1 message every 6 seconds that needs to be changed in order to process a mailing list of 600 or so
members?

Sincerely,


David Moore
Pulp Web Solutions
dhm@pulpwebsolutions.com
731-217-4623


Alex

Posted On: 13 Jul 2006 01:43 PM Hi,




There was nothing contradiced. Phplist with trottle will send out mail over a corse of time. no it will not skip any emails.



Best Regards,
Alex
HostGator Technical Support

Am I the only one who thinks this guy was giving contradicting answers? Or is it me who can't read or understand clear enough. In one reply he says: "And yes if it goes over 200 the emails will bounce back to you and NEVER be sent."

Yet in his next reply he says: " You will need to use the throttle function with phplist to do so. This will send to all 600 just during the corse of a longer peroid of time. You will need a dedicated server to send anymore then 200 per hour. 900+ email lists will need to be done sometime during non peek hours such as 1am - 6am."


Please note that in my original message I stated I had already set up the throttling as specified and he replyed that all over 200 would basically be lost.

Support,


Hello, I am a reseller with Host Gator ( www.pulpwebsolutions.com ). I have a client that is a church. I have created their website (www.northjacksonchurchofchrist.com ). They have asked me to get things setup for a mailing list. They want members of the congregation to be able to subscribe to their weekly bulletin (http://www.northjacksonchurchofchrist.com/subscribe.htm). The church has over 600 members at this time. The weekly bulletin will be sent out as either an html e-mail or as a PDF (yet to be determined). Anyway, my questions are concerning the e-mail policy of host gator, and how is the best way to send out the bulletin to potentially 600+ members, and probably others that are not members.



Currently I have installed PHP List using fantastico and I believe to have
set up the throttling to 200 e-mails per hour at a rate of 1 every 6
seconds. I did this in the config file. My concern of this however is after
reading several posts on the message board is that any number beyond 200
will be dropped and never sent. I have also installed mailman using cPanel
under the e-mail section. Using mailman, I created a mailing listed titled
bulletin@northjacksonchurchofchrist.com . My concern for this program is
basically the same, plus it has not throttle capability.



So, as a reseller, I am asking Host Gator to please inform me of the
appropriate manner to go about sending out around 600 e-mailed church
bulletins once per week. I am sorry to be sending this to support, but
after reading the forums over and over; I seemed to be more confused than enlightened.

[kmaw]

Whoa that was long...

PHPlist will send the first 200 right away, then 200 the second hour and 200 in the third hour... 600 emails sent, just spaced out so you don't go over the 200 limit..

It's suggested you make the setting 175 or something lower than 200 in case there are other emails being sent at the same time.

[bzoeller]

i think you have to set it up where it runs the cron job every 15 minutes or so to process the quene, instead of doing it manually.

[pjlyons]

Now with the upcoming PHPSuExec install - won't this method of throttling emails no longer work? I'm referring to this part of the email I received from HostGator:

If your account is using over 25 processes, 100MB or more per PHP
thread, or exceeds a 30 second execution time, you will receive an
Internal Server Error.

How can we get around this if we have to send out emails to a large group of members?

[riostyles]

I do not use anymore phplist but an other throttler.
Clearly there are two limits on top one of the other:
The sendmail or whatever server process trashing everything over 200 per domain and the throttler eg phplist doing his job.
I recommand to set them to 150 or 160 no more for to let some space for some extra regular mails. What's the matter if the server need's one hour more for to accomplish the mailing.
The 30 secs do not concern this probleme, it is the time limit of the execution of an individual script, including his includes or external calls, excluding the waiting times eg. waiting for a connection, a DB, ...
Some second opinion is welcome.
Doing so I do not have any problem and my script manages some sends over hours.

[apinner]

Does the same 200 emails per hour limit still apply if phplist uses a remote smtp server?

[Serra]

No, only when you are using a local SMTP server. However, if you don't throttle Phplist, you will spike your CPU sending the emails to the remote server and your account will still get suspended.

[apinner]

I decided to ask support, here is what they said.

Me: i want to use phplist for a mailing list, but if i setup phplist to send the mail through a remote server does the restriction apply?
HG: No as long as it does not send mail through our servers

Me: there was also some discussion on the hostgator forums to whether or not emails over the 200 limit were sent with the throttling in place
Me: ie, if i have 600 emails on the list and i throttle it to 200 per hour, will it stop after an hour or send all 600 after 3 hours?
HG: They are not sent and the server gives you an error message if you try to send
HG: It will reject any past 200
Me: it doesn't just space out the 600 over a longer period?
HG: Not at all
Me: and by using a remote smtp with no restrictions is a way around that problem, i could send all 600 with no problems?
HG: Correct that will work

Regarding hardware usage...

Me: would i be likely to run into problems with hardware usage restrictions?
HG: Not that I can tell.

I hope that clears up some of the issues that were raised previously.

[admiral]

What hostgator needs to tell people is, you cant have a mailing list running on any shared or reseller or semi dedicated server hosting option.

having a limit on number of emails per hour defeats the whole point of having a mailing list. If you can not get a dedicated server, then you would need to use one of the several FREE/ad driven web based mailing lists out there.

http://groups.yahoo.com/

http://groups.google.com/

http://lists.topica.com/

http://www.coollist.com/

http://www.bravenet.com/webtools/elist/

these are all free, and have lots of options and do not limit your mailing list emails per hour.

I own and moderate over 40 Mailing lists on YahooGroups and Topica, and most of the groups have over 3000+ members.

[PatongBiz]

I can't get the cronjob to work at all for phplist. When using fantastico it enters the following in the crontab:

cd '/home/userXXX/public_html/list/admin/' ; php -q index.php -p processqueue > /dev/null ;

But that doesn't seem to process the queue. It can't be that I have to have the browser open for the list to finish? As I have more the 2200 members it take some 12 hours to send

[newview]

I'm another who had major problems with the recommended config.php settings with the browser timing out and having to hit the processqueue link every hour. I'm definitely NOT going to do that.

I found that the following settings allow the browser to reload and begin the processqueue every 60 seconds. I was able to send 4 messages of about 1100 each in 4 days.

define("MAILQUEUE_BATCH_SIZE",3);
define("MAILQUEUE_BATCH_PERIOD",60);
define('MAILQUEUE_THROTTLE',19);

So . . . that's 3 emails every 60 seconds, waiting 19 seconds between
emails. That computes to about 187 emails per hour, under the 200/hr limit.

[rmowles]

I found that after the change to phpsuexec I had to change my crontab line to read:

/usr/local/bin/php -q /home/username/public_html/newsletter/admin/index.php -p processqueue > /dev/null;

This uses the older php for processing the queue.

[psican]

I recently installed phplist. When I imported a 2400+ suscribers list I set it to request for user confirmation rather than marking them all as confirmed. Nearly all the confirmation requests were bounced by HG for exceeding the max number of emails sent per hour. So I tried following "Phplist Throttling :: How To" instructions to control the amount of emails sent per hour and stay within the limit. Then I went to "reconcile users" in phplist to resend
the request for confirmation email, and they bounced again!

I have tried changing the batch size and throttle settings so many times I don't even know what they are supposed to be anymore. Now I just have big mess and it's way over my head.

Am I not entering the right number in the right place? Or is it just that the throttle only works for mailing list messages and not for confirmation and system messages? Is there a way to control the speed at which confirmation request emails are sent?

This is how settings are now:
define("MAILQUEUE_BATCH_SIZE",180);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',0);

define('MAILQUEUE_AUTOTHROTTLE',1);
define('USE_DOMAIN_THROTTLE',0);
define('DOMAIN_BATCH_SIZE',1);
define('DOMAIN_BATCH_PERIOD',60);

I'd appreciate a "for dummies" reply, please!

[chaloupe]

Originally Posted by psican

define('MAILQUEUE_THROTTLE',0);



I'd appreciate a "for dummies" reply, please!

Need to change the 0 to 1 if you want to throttle.
define('MAILQUEUE_THROTTLE',1);

Here's how to set it to send 180 email per hour:

define("MAILQUEUE_BATCH_SIZE",180);
define("MAILQUEUE_BATCH_PERIOD",3600);
define('MAILQUEUE_THROTTLE',20);

<<the line just above will add 20 second between each email just to make sure you will not overload. so 1 email is sent every 20 seconds. IF we do the math, 3600 seconds(1 hour) divided by 20 = 180>>

define('MAILQUEUE_AUTOTHROTTLE',0);

We don't need autothrottle since we did our own math with the 180, 3600 and 20.

And wait at least an hour since you can only send 200 email per hour or they will all bounce again.

Best Regards

[psican]

I made the changes suggested by chaloupe, then had PHPList send the confirmation requests again, but it's not throttling them. I'm still getting hundreds of bounce emails from HG for exceeding max emails per hour.

Seems throttling in PHPList does not apply to system messages, only list messages.

[good1]

Wow... so for the past year I have been sending messages to our 700+ members only to find out only a few got through. My questions is, if this is such a monumental problem for HostGator, why don't they automatically throttle ALL outgoing mail to resolve this issue instead of letting us find out after our business is half-ruined?

I have no idea how to do any of the above Fantastico changes, and it sounds like it is not reliable at all.