DOC/Initial Mailman v2.0 with TMDA and MIME filtering HOW-TO

A_and_MIME_filtering_HOW-TO">6.7. Initial Mailman v2.0 with TMDA and MIME filtering HOW-TO

class="anchor" id="line-4">

( http://en.wikipedia.org/wiki/Tagged_Message_Delivery_Agent : TMDA's main difference from other anti-spam systems is the use of a controversial challenge/response system...)

I built the system with Mailman 2.0, TMDA 0.58, Exim v3, mimefilter, mimedecode, and procmail. You don't need mimedecode or mimefilter, but if you want to do mime filtering (even with say strip mime or demime instead) adapting the below is should be simple and obvious. Several aspects and details are going to be specific to the default configurations and installations of the relevant packages under Debian Linux. I will assume that you'll be Linux/Unix savvy enough to both detect those points and work around them as needed.

Background:

  -- I decided to integrate the system via procmail as I also wanted to (easily) add MIME filtering and other forms of message rewriting to my list setup.

  -- A primary goal is that the system operates without requiring either SysAdm or list owner attention.

  -- As discussed on-list One of my wishes was that lists maintain whitelists which are private to them and thus not shared across the entire set of lists serviced by that host.  I will not detail how I went about that except in passing as current TMDA development initiatives will render that much simpler in future.

  -- Elegance was not a goal.  Workability and were.  There are several rough edges left.  Some are noted below.

How-To:

Getting the system working essentially has three steps:

  1) Getting Mailman working transparently under Exim without aliases.

  2) Getting TMDA configured, installed, and tested.

  3) Getting TMDA integrated with Exim and Mailman, using procmail as a glue layer.

You'll be wise to check and test each step as you go along. Finding the correct source of faults with the whole system in place can be a confusing and difficult exercise. Its easier to test and retest as you go along so as to limit the set of possible items that need examination for error.

Step one is fairly simple: Just configure Exim as per the README.Exim or Nigel's HOW-TO on exim.org. Make sure its all working and that Mailman functions correctly in all regards as regards the MTA. You will have to update and extend these configurations later to suit TMDA, but start out making sure you have the base right. If Mailman doesn't work on your system its hardly worth trying to get TMDA also not working.

Install TMDA on your system (Debian uses a $PREFIX of /usr). Make sure you get at least version 0.58. For Debian this means installing from /testing. If you use a later version (such as current CVS) read the changelog carefully as you'll have to make some adaption in your setup to the new features in TMDA. Those changes are left as an exercise for the reader.

Under ~list/ create the following directory hierarchy, with 0700 permissions and owned by list.list (the Debian Mailman user):

    ~list/.tmda ~list/.tmda/cache ~list/.tmda/lists ~list/.tmda/logs ~list/.tmda/filters ~list/.tmda/pending ~list/.tmda/templates

Within that tree create the following empty files, each of zero length, owned by list.list and with 0600 permissions:

    ~list/.tmda/lists/released ~list/.tmda/lists/whitelist.auto ~list/.tmda/lists/blacklist ~list/.tmda/lists/whitelist ~list/.tmda/filters/outgoing

The above files need to be created as otherwise TMDA will error when it fails to find them. If you have particular scaling or performance issues with regard to these address lists you might want to use the dbm or cdb forms instead (see the TMDA documentation for details). If so, make sure you configure ~list/.tmda/config to match.

Next copy the default TMDA message templates to:

    ~list/.tmda/templates/bounce.txt ~list/.tmda/templates/confirm_request.txt ~list/.tmda/templates/confirm_accept.txt

And rewrite them as suitable for your mailing lists.

Now create your TMDA crypt key in:

    ~list/.tmda/crypt_key

and your TMDA configuration in:

    ~list/.tmda/config

My ~list/.tmda/config reads:

    DATADIR = "/var/lib/mailman/.tmda/" MAIL_TRANSFER_AGENT = "exim" DELIVERY = "|/usr/bin/procmail -p -f $SENDER" RECIPIENT_DELIMITER = "+" CRYPT_KEY_FILE = "/var/lib/mailman/.tmda/crypt_key" BARE_APPEND = "/var/lib/mailman/.tmda/lists/whitelist" CONFIRM_APPEND = "/var/lib/mailman/.tmda/lists/whitelist.auto" CONFIRM_MAX_MESSAGE_SIZE = None TEMPLATE_DIR = "/var/lib/mailman/.tmda/templates/" ACTION_INCOMING = "confirm" ACTION_OUTGOING = "bare" FINGERPRINT = ["message-id", "from", "date"] FULLNAME = "My Extremely silly name for my filters that I need to change" HOSTNAME = "the.FQDN.of.my.system" LOGFILE_DEBUG = "/var/lib/mailman/.tmda/logs/tmda_debug.log" LOGFILE_INCOMING = "/var/lib/mailman/.tmda/logs/tmda_incoming.log" PENDING_BLACKLIST_APPEND = "/var/lib/mailman/.tmds/lists/blacklist" PENDING_BLACKLIST_APPEND = "/var/lib/mailman/.tmds/lists/deleted" PENDING_RELEASE_APPEND = "/var/lib/mailman/.tmda/lists/released" PENDING_WHITELIST_APPEND = "/var/lib/mailman/.tmda/lists/whitelist" TIMEOUT = "2w"

You'll specifically need to tailor the values of FULLNAME and HOSTNAME from the above cases.

Note:

  I use a RECIPIENT_DELIMITER of "+" as I use "-" in several list names and don't want the two appearing to run together.  Make sure you read the caveats on using "+" in the TMDA FAQ before chosing the RECIPIENT_DELIMITER you want to use.

In my case ~list/.tmda/logs is actually a symlink to /var/log/TMDA which is owned by list.list and of 0700 permissions and a matching logrotate stanza:

    /var/log/TMDA/procmail.log { monthly compress missingok } /var/log/TMDA/tmda-debug.log { monthly compress missingok } /var/log/TMDA/tmda-incoming.log { monthly compress missingok }

Next, and almost finally is to create the incoming filter with ownership list.list and permissions 0600:

    ~list/.tmda/filters/incoming

The contents should be similar to:

    # # Note: First match wins, so sequence of rules is significant #

    # # Bounce blacklisters #

    from-file ~list/.tmda/lists/blacklist bounce

    # # Explicitly listed whitelist addresses get thru (explicitly because # this is a hand maintained file) #

    from-file ~list/.tmda/lists/whitelist ok

    # # Normal list members get thru. Repeat this line as needed for each # of your lists, replacing "listname" with each listname. #

    from-mailman -attr=members ~list/lists/listname ok

    # # Digest members get thru.  Repeat this line as needed for each of # your lists, replacing "listname" with each listname #

    from-mailman -attr=digest_members ~list/lists/listname ok

    # # Confirmed whitelist addresses get thru (the whitelist.auto file is # built by posters confirming themselves to TMDA) #

    from-file ~list/.tmda/lists/whitelist.auto ok

    # # Mail from the list control gets thru. Repeat these lines as needed # for each of your lists, replacing "listname" with each listname. #

    from listname-admin@myhost.dom ok from listname-owner@myhost.dom ok

Tailor to suit as noted. Soon the complexity (and hand effort of adding the list-specific lines) will no longer be necessary as future versions of TMDA will have a more flexible filter schema that supports variable replacement.

At this point you should be able to test your TMDA configuration (not including the Mailman specific bits), manually injecting messages into tmda-inject while having the environment variables SENDER and RECIPIENT appropriately set and seeing that it doesn't error (check the bounces and ~list/.tmda/logs/tmda-debug.log for specifics on errors).

Check your TMDA setup carefully and pay particular attention to file permissions and ownership errors. Make sure you do all your testing as the Mailman user ("list" for Debian).

You're now ready for the last step: integrating TMDA with Exim via procmail. Place the procmail rc in:

  ~list/.procmailrc

Again, the ownership is list.list and permissions 0600. It should read something like:

    VERBOSE=off SHELL=/bin/sh PATH=/usr/bin:/usr/local/bin:/usr/lib:/usr/local/lib LOGFILE=/var/log/TMDA/procmail.log MAILDIR=/var/lib/mailman/locks

    # # Commands don't get filtered #

    :0 w * ACTION ?? mailcmd |/var/lib/mailman/mail/wrapper $ACTION $TARGET

    # # Pass bounces thru directly. #

    :0 w * ACTION ?? mailowner * ^Return-path: &lt;&gt; | /var/lib/mailman/mail/wrapper $ACTION $TARGET

    # # Mail that has already been filtered by TMDA.  This is an added # header that is used to check for messages which have already been # processed by this TMDA/Mailman procmail filter. #

    :0 w * $^X-Mail-Filter-Recipient: $RECIPIENT {

      # # Send the message to Mailman for processing.  This is a separate # recipe to make the extensions for MIME filter etc below clearer # and simpler. #

      :0 w * ACTION ?? post | /var/lib/mailman/mail/wrapper $ACTION $TARGET

      :0 w * ACTION ?? mailowner | /var/lib/mailman/mail/wrapper $ACTION $TARGET

    # # Add the flag header so we can detect messages coming out of TMDA, # even as distinct from mail from other TMDA users. #

    :0 fw | /usr/bin/formail -A "X-Silly-Me-Mail-Filter-Recipient: $RECIPIENT"

    # # Everything else (-admin, -owner, and posts which haven't been # passed by TMDA) get filtered thru tmda-filter.  If you want # list-specific inbound filters instead globally shared lists, # insert a small script here which writes a message customised # inbound filter in (say) /tmp/tmda-inbound.$$, call tmda-filter # with a matching -I argument, and then delete the temporary inbound # filter afterwards. #

    :0 w | /usr/bin/tmda-filter -c /var/lib/mailman/.tmda/config

    # # Take the exit code from TMDA.  This allows procmail to return in # error to the MTA. #

    EXITCODE=$?

Notes:

  1) Please use a different loop detection header than X-Silly-Me-Mail-Filter-Recipient.  Make something up that fits your needs.

  2) The added loop detection will not be necessary with TMDA versions after 0.58 as they add their own custom header which you can use for host and recipient specific loop detection.

As I do want to do MIME and other message pre-processing, and I want the ability to send messages which bypass the MIME filters, my ~list/.procmailrc is a little more complex. Adapt as appropriate to your installation:

    VERBOSE=off SHELL=/bin/sh PATH=/usr/bin:/usr/local/bin:/usr/lib:/usr/local/lib LOGFILE=/var/log/TMDA/procmail.log MAILDIR=/var/lib/mailman/locks

    # # Commands don't get filtered #

    :0 w * ACTION ?? mailcmd |/var/lib/mailman/mail/wrapper $ACTION $TARGET

    # # Pass bounces thru directly. #

    :0 w * ACTION ?? mailowner * ^Return-path: &lt;&gt; | /var/lib/mailman/mail/wrapper $ACTION $TARGET

    # # Mail that has already been filtered by TMDA.  This is an added # header that is used to check for messages which have already been # processed by this TMDA/Mailman procmail filter. #

    :0 w * $^X-XXX-Filter-Recipient: $RECIPIENT {

      # # Posts sent to a list #

      :0 w * ACTION ?? post {

        # # Got an X-MimeStripNo header?  Don't MIME strip #

        :0 w * ^X-MimeStripNo {

          # # Remove the X-MimeStripNo header #

          :0 fw | formail -I X-MimeStrip

        :0 w * ! ^X-MimeStripNo {

          # # Mimefilter -- As mimefilter requires that the local directory # be writable and that various environment variables are defined, # we use a wrapper script. #

          :0 fw | ~archiver/bin/MimeFilter

          # #  Get rid of quoted-printable #

          :0 fw | /usr/bin/mimedecode }

        # # If there's nothing left of the message (less than 20 bytes), don't # bother sending it forward to Mailman. #

        :0 wb * &lt; 20 | /dev/null

        # # We have a post! #

        :0 w | /var/lib/mailman/mail/wrapper $ACTION $TARGET

      # # Messages to -admin and -ower #

      :0 w * ACTION ?? mailowner | /var/lib/mailman/mail/wrapper $ACTION $TARGET

    # # Add the flag header so we can detect posts coming out of TMDA, # even as distinct from mail from other TMDA users. #

    :0 fw | /usr/bin/formail -A "X-XXX-Mail-Filter-Recipient: $RECIPIENT"

    # # Everything else (-admin, -owner, and posts) get filtered thru TMDA #

    :0 w | /usr/bin/tmda-filter -c /var/lib/mailman/.tmda/config

    # # Take the exit code from TMDA.  This allows procmail to return in # error to the MTA. #

    EXITCODE=$?

Yeah, I could make that procmailrc cleaner and simpler. It works. Its a 1.0. I don't argue with working 1.0 versions.

The contents of ~archiver/bin/MimeFilter are:

    #!/bin/bash #set -x

    # # Syntax: # #   MimeStrip # # Author: J C Lawrence &lt;claw@kanga.nu&gt; #

    # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= # Config (Edit this bit to suit your setup!) # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

    # Where is mimefilter?

    mimefilter="/usr/bin/mimefilter"

    # Note: Make sure that the value of tmpdir is writeable by the UID or GID # that this script will execute under.

    tmpdir="/tmp/mimestrip.${$}"

    # Do you want to save an mbox of the messages before filtering # in $savefile_pre?   (0/1)

    feature_savepre=1

    # Do you want to save an mbox of the messages after filtering # in $savefile_post?  (0/1)

    feature_savepost=1

    # File to save pre-filtered messages in

    savefile_pre="/tmp/mimestrip.prefilter"

    # File to same post filtered messages in

    savefile_post="/tmp/mimestrip.postfilter"

    # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= # MimeFilter setup (Don't edit from here down) # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

    # The name of the mailing list this message is intended for.  Used # as the return address of the warning issued to the orginal author # if the message is not already clean.

    export list=${USER}

    # The address of the mailing list this message is intended for. # Used in the X-Loop field of the warning issued to the original # author if the message is not already clean.

    export listaddr=${USER}@${DOMAIN}

    # The administrative (owner) address of the mailing list this # message is inteded for.  Used in the return address of the warning # issued to the original author if the message is not already clean.

    export listreq=${USER}-admin@${DOMAIN}

    # The email address of the maintainer of the mailing list this # message is inteded for.  If it is defined, it is used to send the # maintainer original carbon copies of messages that have been # modified by this filter -- if filter_mime_cc_maintainer is # affermative, of course.

    export maintainer=${USER}-owner@${DOMAIN}

    # A boolean flag: if affermative (i.e., if it matches the /y/i Perl # regular expression), the mimefilter script will send carbon copies # of every cleaned (modified) message to the maintainer of the # mailing list the message is intended for.

    export filter_mime_cc_maintainer=${copy_maintainer}

    # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= # Start working # -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

    # Build a safe working directory for mimefilter.  We need to do this as # postfix by default will execute this script in /var/spool/postfix and # mimefilter will try and create temp directories and files under there, # which will fail on permissions.

    # Note: Make sure that the value of tmpdir is writeable by the UID or GID # that this script will execute under.

    /bin/rm -rf ${tmpdir} 2&gt; /dev/null /bin/mkdir ${tmpdir} cd ${tmpdir}

    /bin/cat | ${mimefilter}

    # Save copies of the before and after messages off to a temp file for # analysis/debugging

    if [ ${feature_savepre} -eq 1 ] then /bin/cat ${holdfile} &gt;&gt; ${savefile_pre} echo &gt;&gt; ${savefile_pre} fi

    if [ ${feature_savepost} -eq 1 ] then /bin/cat ${workfile} &gt;&gt; ${savefile_post} echo &gt;&gt; ${savefile_post} fi

    # Clean up

    /bin/rm -rf ${tmpdir} exit

Most of all that hassle is to get the environment variables mimefilter needs properly set and get the current working directory set to something I've write permissions to (eg /tmp). I could have done all that inside procmail (and probably should have). I didn't for purely historical reasons.

mimedecode is a little tool which flattens quoted-printable into US ASCII. I find it rather useful. Take care here if you have posters posting with other character sets.

As always: caveat emptor.

At this point you should be able to test your procmail and TMDA configuration my injecting messages into procmail as below:

    # List posts cat message_file | procmail -mpt ACTION='post' TARGET='listname' ~list/.procmailrc

    # -request commands cat message_file | procmail -mpt ACTION='mailcmd' TARGET='listname' ~list/.procmailrc

    # -admin and -owner messages cat message_file | procmail -mpt ACTION='mailowner' TARGET='listname' ~list/.procmailrc

Test it all out. Make sure TMDA is properly holding messages. Release, whitelist, blacklist etc them via:

  tmda-pending -c ~list/.tmda/config

and watch the contents of ~list/.tmda/lists to ensure that the right things are happening.

Work the whole system until you're certain that everything in the procmailrc and your TMDA configuration has been exercised and shown correct.

Don't forget to go back and delete all your test addresses from the files under ~list/.tmda/lists. Remember to NOT delete the files and to do all your testing as the list user (file permissions are critical).

You're now ready for the last step: Integrating with your MTA. Before we get to the message details some discussion on why I used Exim for this:

  I started out wanting and trying to use Postfix.  The core problem I hit was making the filter process hung off the alias execute as the right user.  I needed it to execute as the list user so it would be able to read the Mailman list configurations to verify subscribers from there.  If you want to entirely separate posting rights from list subscription this wouldn't be a problem (just remove the mailman bits from ~list/filters/incoming).

  Getting the filter process to execute as 'list' under postfix proved a bitch.  Its not impossible, but it requires either messing with content filters (which have a high performance overhead for ALL messages) or doing a vtable setup with a custom transport, which requires that you run your lists from a sub-domained virtual host (eg lists.kanga.nu).  I am unwilling to make either compromise.  YMMV.

  Exim conversely makes defining the specific effective UID and GID for an alias filter process very easy.  I also like, know, and use Exim on other systems, so moving my list server back to Exim was no great pain.

  I don't like and refuse to use either Sendmail, and so can't comment on how TMDA may be integrated with it.  I'm also not a QMail user and so can't comment for similar reasons.  For those familiar with those respective MTAs you should be able to base off the below recipes (with the above caveats) successfully.

The full set of Mailman-specific sections needed in /etc/exim/exim.conf are as below. Compare them carefully with the Nigel's HOW-TO for basic Exim/Mailman integration. The changes aren't large but you should understand each of them. The documentation at exim.org is excellent.

These configurations are mildly Exim v3 specific. Exim v4 is noticeably but not fundamentally different. Use the documentation at exim.org to guide you thru the differences.

  In the main configuration section:

    # Add the Mailman user (list) to the value of trusted users so # procmail can use -p to pass environment variables through # (required by TMDA)

    trusted_users = mail:list

    # $HOME dir for Mailman

    MAILMAN_HOME=/var/lib/mailman

    # Wrapper script for Mailman

    MAILMAN_WRAP=MAILMAN_HOME/mail/wrapper

    # UID and GID for Mailman

    MAILMAN_UID=list MAILMAN_GID=mail

  In the transports section:

    list_transport: driver = pipe command = /usr/bin/procmail -mpt ACTION='post' TARGET='${lc:$local_part}' MAILMAN_HOME/.procmailrc # command = MAILMAN_WRAP post ${lc:$local_part} current_directory = MAILMAN_HOME home_directory = MAILMAN_HOME user = MAILMAN_UID group = MAILMAN_GID return_path_add delivery_date_add envelope_to_add environment = EXTENSION=${substr_1:$local_part_suffix}:\ RECIPIENT=$local_part$local_part_suffix@$domain

    list_request_transport: driver = pipe command = /usr/bin/procmail -mpt ACTION='mailcmd' TARGET='${lc:$local_part}' MAILMAN_HOME/.procmailrc # command = MAILMAN_WRAP mailcmd ${lc:$local_part} current_directory = MAILMAN_HOME home_directory = MAILMAN_HOME user = MAILMAN_UID group = MAILMAN_GID return_path_add delivery_date_add envelope_to_add environment = EXTENSION=${substr_1:$local_part_suffix}:\ RECIPIENT=$local_part$local_part_suffix@$domain

    list_admin_transport: driver = pipe command = /usr/bin/procmail -mpt ACTION='mailowner' TARGET='${lc:$local_part}' MAILMAN_HOME/.procmailrc # command = MAILMAN_WRAP mailowner ${lc:$local_part} current_directory = MAILMAN_HOME home_directory = MAILMAN_HOME user = MAILMAN_UID group = MAILMAN_GID return_path_add delivery_date_add envelope_to_add environment = EXTENSION=${substr_1:$local_part_suffix}:\ RECIPIENT=$local_part$local_part_suffix@$domain

  In the directors section:

    list_owner_director: driver = smartuser require_files = MAILMAN_HOME/lists/${lc:$local_part}/config.db suffix = "-owner" new_address = "${lc:$local_part}-admin@${domain}"

    owner_list_director: driver = smartuser require_files = MAILMAN_HOME/lists/${lc:$local_part}/config.db prefix = "owner-" new_address = "${lc:$local_part}-admin@${domain}"

    list_admin_director: driver = smartuser suffix = -admin require_files = MAILMAN_HOME/lists/${lc:$local_part}/config.db transport = list_admin_transport

    list_request_director: driver = smartuser suffix = -request require_files = MAILMAN_HOME/lists/${lc:$local_part}/config.db transport = list_request_transport

    list_director: driver = smartuser require_files = MAILMAN_HOME/lists/${lc:$local_part}/config.db transport = list_transport suffix = +* suffix_optional

Pay particular attention to the value of "suffix" in list-director and make sure it agrees with RECIPIENT_DELIMITER in ~list/.tmda/config.

You should now be up and running with TMDA integrated as the front face of your Mailman lists. If you want to monitor your TMDA pending queue for your Mailman installation you can periodically run

  tmda-pending -c ~list/.tmda/config

Or just drop the following cronjob in the Mailman user's crontab:

  * * * * * /usr/bin/tmda-pending -CTb | mail you@domain.tld

to be emailed regular status updates on new messages in TMDA pending.

Converted from the Mailman FAQ Wizard

This is one of many Frequently Asked Questions.

MailmanWiki: DOC/Initial Mailman v2.0 with TMDA and MIME filtering HOW-TO (last edited 2015-01-31 02:36:58 by msapiro)