Differences between revisions 4 and 5
Revision 4 as of 2008-07-31 14:02:55
Size: 4064
Editor: dunxd
Comment:
Revision 5 as of 2008-07-31 14:02:56
Size: 4121
Editor: dunxd
Comment: Migrated to Confluence 4.0
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#pragma page-filename DOC/versions/14352603 #pragma page-filename DOC/versions/16941082
Line 3: Line 3:
Line 11: Line 12:
(From [[http://mail.python.org/pipermail/mailman-users/2001-May/011257.html]])
(From [[http://mail.python.org/pipermail/mailman-users/2001-May/011257.html|http://mail.python.org/pipermail/mailman-users/2001-May/011257.html]])
Line 33: Line 35:
As qrunner processes stuff, it deletes those files. So as new stuff comes in, they get stored as close to the start of the directory inode as possible, so when qrunner quits and restarts, if it hasn't processed everything in the directory, it starts over with newer stuff, leaving older stuff deep into the inode -- and it'll never GET to that stuff deep in the inode until the system quiets down and it's given a change to catch up. That means something could literally stay in the queue forever if the system never slows down enough to allow qrunner to clean up. As qrunner processes stuff, it deletes those files. So as new stuff comes in, they get stored as close to the start of the directory inode as possible, so when qrunner quits and restarts, if it hasn't processed everything in the directory, it starts over with newer stuff, leaving older stuff deep into the inode and it'll never GET to that stuff deep in the inode until the system quiets down and it's given a change to catch up. That means something could literally stay in the queue forever if the system never slows down enough to allow qrunner to clean up.
Line 35: Line 37:
By extending qrunner's lifetime, you're allowing it to go much deeper into the inode. It's STILL not FIFO (and this is why replies get seen before messages being replied to, and why digests have messages scrambles; Barry and I have talked at length about this and the queueing should be FIFO in 2.1....), but you're a lot less likely to have a given message buried in the queue for hours. It doesn't fix the problem -- but significantly (from my system's operations) reduces it, and limits the impact when it does happen. By extending qrunner's lifetime, you're allowing it to go much deeper into the inode. It's STILL not FIFO (and this is why replies get seen before messages being replied to, and why digests have messages scrambles; Barry and I have talked at length about this and the queueing should be FIFO in 2.1....), but you're a lot less likely to have a given message buried in the queue for hours. It doesn't fix the problem but significantly (from my system's operations) reduces it, and limits the impact when it does happen.
Line 37: Line 39:
If you find you still have stuff waiting around -- after you make this change, if you're still running 2 hours behind, extend it to 3 hours. But be aware that if for some reason qrunner decides to wedge, you're locking up your system for longer periods of time. Keeping an eye on things is good. If you find you still have stuff waiting around after you make this change, if you're still running 2 hours behind, extend it to 3 hours. But be aware that if for some reason qrunner decides to wedge, you're locking up your system for longer periods of time. Keeping an eye on things is good.
Line 41: Line 43:
''Last changed on Tue Sep 5 23:03:14 2006 by'' Brad Knowles
<<Color2(Converted from the Mailman FAQ Wizard, col=darkgreen)>>This is one of many [[../Frequently Asked Questions|Frequently Asked Questions]].
''Last changed on Tue Sep 5 23:03:14 2006 by'' Brad Knowles<<BR>> Converted from the Mailman FAQ Wizard

This is one of many [[../Frequently Asked Questions|Frequently Asked Questions]].

6.6. Mailman Performance Tuning for Mail Delivery

These are generic performance tuning tips - for tips for specific MTAs see the other FAQs in this section. This stuff is likely to be Mailman 2.0 specific.

General points:-

(From http://mail.python.org/pipermail/mailman-users/2001-May/011257.html)

A few things to check...

Make sure your batch size is small:

 SMTP_MAX_RCPTS = 10

Set your qrunner proc to live longer, and extend the lock life:

 QRUNNER_LOCK_LIFETIME = hours(10)
 QRUNNER_PROCESS_LIFETIME = minutes(15)
 QRUNNER_MAX_MESSAGES = 300

Set these to 20 hours, 2 hours and 50000

Here's why: qrunner doesn't process the queue FIFO. Instead, it opens up the directory and processes the entries sequentially. This implies that if you get a lot of stuff in the queue, qrunner will quit after 15 minutes and start over.

As qrunner processes stuff, it deletes those files. So as new stuff comes in, they get stored as close to the start of the directory inode as possible, so when qrunner quits and restarts, if it hasn't processed everything in the directory, it starts over with newer stuff, leaving older stuff deep into the inode – and it'll never GET to that stuff deep in the inode until the system quiets down and it's given a change to catch up. That means something could literally stay in the queue forever if the system never slows down enough to allow qrunner to clean up.

By extending qrunner's lifetime, you're allowing it to go much deeper into the inode. It's STILL not FIFO (and this is why replies get seen before messages being replied to, and why digests have messages scrambles; Barry and I have talked at length about this and the queueing should be FIFO in 2.1....), but you're a lot less likely to have a given message buried in the queue for hours. It doesn't fix the problem – but significantly (from my system's operations) reduces it, and limits the impact when it does happen.

If you find you still have stuff waiting around – after you make this change, if you're still running 2 hours behind, extend it to 3 hours. But be aware that if for some reason qrunner decides to wedge, you're locking up your system for longer periods of time. Keeping an eye on things is good.

In addition, if you had serious problems with your queues at some time in the distant past, you may want to consider stopping Mailman, then moving aside all the affected queues and re-creating them with the same ownership and permissions. The problem is that most OSes never shrink the size of their directories, so if they have to expand them to accomodate more files at a given point in time, they will permanently be left at that expanded size. Even if most of the directory entries are empty, it still takes time to scan through all them to make sure that everything has been processed. So, it makes sense to go in and periodically trim all the queue directories down.

Last changed on Tue Sep 5 23:03:14 2006 by Brad Knowles
Converted from the Mailman FAQ Wizard

This is one of many Frequently Asked Questions.

MailmanWiki: DOC/Mailman Performance Tuning for Mail Delivery (last edited 2015-01-31 02:36:58 by msapiro)