4.39. HELP! Mailman is munging HTML & MIME-formatted messages before they are sent out? (problems with Mailman 2.1.x footers)

a.k.a., "Why are footers (or message bodies) being shown as attachments to HTML-formatted messages?"

For Mailman 2.0.x, see also 4.05 Why don't the footers show up on some list messages? (Mailman 2.0.x).

The problem is that there is no general-purpose method of determining where a footer should be properly attached to a message that is HTML or MIME formatted. To figure that out for HTML-formatted messages, you'd have to implement a complete HTML imaging engine, so that you could see what was the apparent "bottom" of the page (as it appears to the user), and then work backwards from there to figure out where to perform your surgery.

Doing so would also lock you into a solution that only works for those MUAs which implement the same (or compatible) HTML engines, and would almost certainly make the situation worse for everyone else.

So, what Mailman does instead is to take the entire HTML message as sent, put that into a MIME bodypart, add another MIME bodypart for the footer, and then wrap all that up and send it out as a message that is formatted as a different kind of MIME message. Some MUAs will display the result in a fashion reasonably close to the format in which the message was submitted, and some won't. Those that have problems may result in the footer being displayed as an attachment, or worse. The same issue applies to msg_header, if any, although in this case, some MUA's may display the header as the message and the body as an attachment.

Note that this problem is not unique to HTML-formatted messages. Messages that have MIME multipart/mixed or multipart/alternative bodyparts and signed messages have the same problem - Mailman can't figure out where it could safely append the footers to the existing bodyparts, so it has to wrap the whole message up in another MIME structure and add a bodypart at the bottom for the footers. Even if the message is formatted as text/plain, it may have a different charset or content-transfer-encoding than Mailman would use for the footers. In that case, the incoming text/plain message would still get wrapped up as one bodypart with the footers created as a second bodypart, and the whole thing sent out as a multipart/mixed.

To solve this problem, your options are:

 1. Configure Mailman to remove the msg_header/msg_footer
 2. Configure Mailman to strip all HTML and MIME formatting and send out all
      messages as text-only (text/plain)
 3. Get everyone to change the MUA they use to one that is more HTML/MIME-aware
 4. Live with the problem

Note that this issue is not specific to a particular version of Mailman. This is a general problem with messages that are formatted in HTML or MIME, which is one reason why Mailman supports methods of stripping and/or converting HTML and MIME to more readable formats.

Keep in mind that the default Mailman content filtering includes more than just text/plain. If you change the configuration to strip everything but text/plain, you will break cryptographic signatures from S/MIME and PGP/MIME MUAs, and will probably cause additional problems for your users. Choosing option #2 from the above list is quite a bit more difficult to configure than you may think it is. Even if you can get the system to do this, the result would most likely be quite a bit more drastic than you think. In recent Mailman there is a scrub_nondigest list setting to flatten the message to plain text and store non plain text parts aside and replace them with links as is done for the plain text format digest. Setting this On will help, but may not be acceptable for other reasons.

There are many examples of this issue being discussed in the archives, with the threads at http://mail.python.org/pipermail/mailman-users/2004-April/036378.html and http://mail.python.org/pipermail/mailman-users/2004-July/038003.html being two examples.

Remember - this is not an HTML problem. This is a MIME and MIME/HTML problem. All MIME-formatted messages may have the same problem.

Patching Mailman to solve this problem

(Note that the patch described in this section was implemented against version 2.1.5 of Mailman and does not apply cleanly to the current stable Mailman release.  It is not known whether the authors of this patch have released an updated version.)

A note from a mailman user:

We have solved this problem (in a way that seems to mostly work), and implemented these changes for our install of mailman 2.1.5. Information about what we did can be found here:

http://mail.python.org/pipermail/mailman-developers/2005-February/017850.html

What we developed works very well for us.

If you need this functionality too, and would like to fix our changes to incorporate in the mailman codebase, please feel free to use what we have done. Implementing these changes took quite a bit of research, so this will save you some time.

Using mimedefang to solve this problem

Jonathan Kamens has implemented and released a solution to this problem using mimedefang.  His solution scans messages for Mailman footers and moves them from separate attachments into the text and/or HTML body parts of the messages, using the logic already built into mimedefang for this purpose.  This functionality is keyed off of special tokens at the beginning and end of the msg_footer setting for each mailing list, which means that (a) it is general-purpose and does not require hard-coding list names, footer text, etc. directly in the mimedefang filter script, and (b) it is only activated for lists that actually want to use it.

For more information, see http://stuff.mit.edu/~jik/software/mailman_mimedefang/ .

Converted from the Mailman FAQ Wizard

This is one of many Frequently Asked Questions.