6.6. Mailman Performance Tuning for Mail Delivery

These are generic performance tuning tips - for tips for specific MTAs see the other FAQs in this section. This stuff is likely to be Mailman 2.0 specific.

General points:-

• Disk throughput is the primary driver for any mail system, so don't cheat on your disk config

• DNS lookups will be needed in bulk by your MTA - put a local DNS cache on the machine itself See DNS performance FAQ entry at Improving performance by local DNS caching.

• choose an MTA that you can handle and fits your needs (postfix, sendmail, and exim are the ones most often recommended on the list)

• Make sure you understand your OS well enough to be able to do basic system performance tuning, such as discussed in FAQ 4.56 at 4.56 What's with all these Mailman-python-qrunner processes? Why are they eating up all the memory on my server? (performance-tuning tips).

(From http://mail.python.org/pipermail/mailman-users/2001-May/011257.html)

A few things to check...

Make sure your batch size is small:

 SMTP_MAX_RCPTS = 10

Set your qrunner proc to live longer, and extend the lock life:

 QRUNNER_LOCK_LIFETIME = hours(10)
 QRUNNER_PROCESS_LIFETIME = minutes(15)
 QRUNNER_MAX_MESSAGES = 300

Set these to 20 hours, 2 hours and 50000

Here's why: qrunner doesn't process the queue FIFO. Instead, it opens up the directory and processes the entries sequentially. This implies that if you get a lot of stuff in the queue, qrunner will quit after 15 minutes and start over.

As qrunner processes stuff, it deletes those files. So as new stuff comes in, they get stored as close to the start of the directory inode as possible, so when qrunner quits and restarts, if it hasn't processed everything in the directory, it starts over with newer stuff, leaving older stuff deep into the inode – and it'll never GET to that stuff deep in the inode until the system quiets down and it's given a change to catch up. That means something could literally stay in the queue forever if the system never slows down enough to allow qrunner to clean up.

By extending qrunner's lifetime, you're allowing it to go much deeper into the inode. It's STILL not FIFO (and this is why replies get seen before messages being replied to, and why digests have messages scrambles; Barry and I have talked at length about this and the queueing should be FIFO in 2.1....), but you're a lot less likely to have a given message buried in the queue for hours. It doesn't fix the problem – but significantly (from my system's operations) reduces it, and limits the impact when it does happen.

If you find you still have stuff waiting around – after you make this change, if you're still running 2 hours behind, extend it to 3 hours. But be aware that if for some reason qrunner decides to wedge, you're locking up your system for longer periods of time. Keeping an eye on things is good.

In addition, if you had serious problems with your queues at some time in the distant past, you may want to consider stopping Mailman, then moving aside all the affected queues and re-creating them with the same ownership and permissions. The problem is that most OSes never shrink the size of their directories, so if they have to expand them to accomodate more files at a given point in time, they will permanently be left at that expanded size. Even if most of the directory entries are empty, it still takes time to scan through all them to make sure that everything has been processed. So, it makes sense to go in and periodically trim all the queue directories down.

Converted from the Mailman FAQ Wizard

This is one of many Frequently Asked Questions.