Differences between revisions 8 and 22 (spanning 14 versions)
Revision 8 as of 2009-01-02 16:12:49
Size: 7299
Editor: p@state-of-mind
Comment: More formatting
Revision 22 as of 2009-11-29 10:51:39
Size: 8651
Editor: barry
Comment: Migrated to Confluence 4.0
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
#pragma page-filename DEV/versions/7602188
= LMTP in Mailman =
[[http://www.faqs.org/rfcs/rfc2033.html|RFC 2033, Local Mail Transport Protocol (LMTP)]] provides Mailman with a unique opportunity to provide a better user experience when accepting initial postings. Two improvements in the process are available with LMTP.
#pragma page-filename DEV/versions/3604512
= LMTP in Mailman 3 =
Line 5: Line 4:
First, we can eliminate most of the integration cruft we currently have with supporting multiple MTAs for incoming mail. Most of the major SMTP servers support LMTP delivery, so by providing an LMTP server in Mailman, the hope is that we can avoid all the crufty MTA-specific alias hackery. [[http://www.faqs.org/rfcs/rfc2033.html|RFC 2033, Local Mail Transport Protocol (LMTP)]] provides Mailman with a unique opportunity to provide a better user experience when accepting initial postings. Two improvements in the process are available with LMTP.
Line 7: Line 6:
Second, and perhaps more importantly, we can support better anti-backscatter and anti-spam defenses for messages sent to the mailing lists, by rejecting messages in the SMTP dialog instead of having to make the determination way later and sending a bounce message. Ian Eiloart describes [[http://mail.python.org/pipermail/mailman-developers/2008-March/019956.html|what is possible]]: First, we can eliminate most of the integration cruft we currently have with supporting multiple MTAs for incoming mail. Most of the major SMTP servers support LMTP delivery, so by providing an LMTP server in Mailman, the hope is that we can avoid all the crufty MTA-specific alias hackery.
Line 9: Line 8:
{{{
on connect:
accept the connection
HELO/EHLO:
reject if the sending MTA isn't known
MAIL FROM:
accept (perhaps unless the sender address is forbidden to post to all
lists).
RCPT TO:
accept if the sender has permissions to post to the list, otherwise reject.
This is the last chance to give a list specific response to an MTA that is
engaged in a callout.
DATA:
reject null senders here if appropriate. Rejecting a null sender at RCPT TO
or earlier might break callouts.
.............
.
Check the data, reject if inappropriate for a specific list (but this is
likely to cause a bounce from our MTA). Because we've decided to trust the
sender, we should be OK to bounce a message here, unless the list is an
open list.
}}}
Second, and perhaps more importantly, we can support better anti-backscatter and anti-spam defenses for messages sent to the mailing lists, by rejecting messages in the SMTP dialog instead of having to make the determination way later and sending a bounce message. Ian Eiloart describes [[http://mail.python.org/pipermail/mailman-developers/2008-March/019956.html|what is possible]].
Line 32: Line 10:
I believe MM3's architecture can easily support this, and I've been working with Ian on the mailing list to sketch out a design. I think we should do this for MM3. == How to set up LMTP delivery for Mailman 3 ==
Line 34: Line 12:
Ian also describes a useful set of enhanced error codes:

{{{
X.1.1 Bad destination mailbox address
X.2.4 Mailing list expansion problem
X.5.3 Too many recipients
X.7.2 Mailing list expansion prohibited
The sender is not authorized to send a message to the intended mailing list.

X is 4 for a temporary error, 5 for a permanent error.
}}}

For details see:

 * [[http://www.faqs.org/rfcs/rfc1893.html|RFC 1893 - Enhanced Mail System Status Codes]]
 * [[http://www.faqs.org/rfcs/rfc2034.html|RFC 2034 - SMTP Service Extension for Returning Enhanced Error Codes]]
== LTMP with various MTAs ==
Line 54: Line 15:
Postfix uses the Postfix lmtp client to transport messages to a LMTP server. The defaults for this client are configured in master.cf:
{{{#!wiki tip

Ubuntu servers

 
If you're getting `Connection refused` errors, you might be hitting [[https://bugs.launchpad.net/ubuntu/+source/dhcp3/+bug/340383/|bug 340383]]. Try commenting out the 127.0.1.1 setting in your `/etc/hosts` file. Because Postfix runs under a chroot on Ubuntu, you'll also need to comment this out in `/var/spool/postfix/etc/hosts`.
}}}

Postfix uses the Postfix `lmtp` client to transport messages to a LMTP<<BR>> server. You probably already have this in your Postfix `master.cf` file:
Line 57: Line 27:
 1. ==========================================================================
 1. service type  private unpriv  chroot  wakeup  maxproc command + args
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
Line 60: Line 30:

 1.
==========================================================================
# ==========================================================================
Line 67: Line 36:
==== Transporting messages to mailman LMTP server ====
Create a transport(5) type table e.g. /etc/postfix/mailman_lists and add each mailman mailing list address on a single line as key to the left hand side of the transport table. For each key specify the destination of the LMTP server as value.
We're going to use a transport map to tell Postfix to deliver all messages<<BR>> destined for a Mailman mailing list to its lmtp service. Fortunately, Mailman<<BR>> is already able to write the correct transport map based on your configuration<<BR>> settings. All you need to do is add a `transport_maps` setting in your<<BR>> Postfix `main.cf` file:
Line 70: Line 38:
{{{#!wiki warning
Always specify mail addresses as FQDN mail addresses. If not the trivial-rewrite agent might "fix" an imcomplete mail address adding $myorigin, which might result in a wrong destination.
{{{
transport_maps =
    ...
    hash:/path/to/mailman/var/data/postfix_lmtp
    ...
Line 74: Line 45:
Now Postfix knows where it should transport messages to, but it doesn't know<<BR>> yet it should accept messages for the given recipients - it doesn't know the<<BR>> recipients are valid recipients. You can reuse the Mailman generated<<BR>> transport map for this by adding the following to your `main.cf` file:
Line 75: Line 47:
{{{#!wiki caution
A destination consists of the service name, the hostname or IP address and the (optional) port each separated by a colon. If the optional LMTP port is omitted the Postfix lmtp client will use $lmtp_tcp_port (default: 24).
{{{
local_recipient_maps =
    ...
    hash:/path/to/mailman/var/data/postfix_lmtp
    ...
Line 79: Line 54:
Here's an example of /etc/postfix/mailman_lists: ==== Virtual domain ====

If the mailman list addresses are part of `$virtual_alias_domains` or `$virtual_mailbox_domains` add `postfix_lmtp` to the listing of `$virtual_alias_maps`:
Line 82: Line 59:
 1. key                           value
mailman@example.org             lmtp:localhost
mailman-admin@example.org       lmtp:localhost
mailman-bounces@example.org     lmtp:localhost
mailman-confirm@example.org     lmtp:localhost
mailman-join@example.org        lmtp:localhost
mailman-leave@example.org       lmtp:localhost
mailman-owner@example.org       lmtp:localhost
mailman-request@example.org     lmtp:localhost
mailman-subscribe@example.org   lmtp:localhost
mailman-unsubscribe@example.org lmtp:localhost
virtual_alias_maps =
    ...
    hash:/path/to/mailman/var/data/postfix_lmtp
    ...
Line 95: Line 65:
Once all entries have been added to the /etc/postfix/mailman_lists file use the Postfix postmap command to create a database: ==== Relay domain ====

If the mailman list addresses are part of the relay domain namespace add `postfix_lmtp` to the listing of `$relay_recipient_maps`:
Line 98: Line 70:
# postmap /etc/postfix/mailman_lists
}}}

Then add the database to the list of $transport_maps in Postfix' main.cf configuration file:

{{{
transport_maps:
relay_recipient_maps =
Line 106: Line 72:
    hash:/etc/postfix/mailman_lists     hash:/path/to/mailman/var/data/postfix_lmtp
Line 113: Line 79:
For a dedicated list server e.g. list.example.com simply add the hostname as key to a transport map and specify the mailman LMTP server as destination:
For a dedicated list server e.g. ''list.example.com'' simply add the hostname as key to a transport map and specify the mailman LMTP server as destination:
Line 116: Line 83:
 1. key                           value
list.example.org                lmtp:localhost
# key                           value
list.example.org                lmtp:inet:localhost
Line 120: Line 87:
Be aware though that Postfix has no knowledge of valid recipients in the destination's (sub)domain (here: list.example.org) if you specify the list server as noted above. Postfix will accept and transport any message destined for list.example.org to the mailman LMTP server and it will be the LMTP servers task to reject messages for invalid or non-existing recipients. Be aware though that Postfix has no knowledge of valid recipients in the destination's (sub)domain (here: list.example.org) if you specify the list server as noted above. Postfix will accept and transport any message destined for ''list.example.org'' to the mailman LMTP server and it will be the LMTP servers task to reject messages for invalid or non-existing recipients.
Line 122: Line 89:
There are two ways to prevent putting such load on the mailman LMTP server. First specify each valid recipient address in a transport table as shown in the initial example. Secondly use the reject_unverified_recipient option and let the Postfix verify(8) daemon find out if the recipient address exists. This way Postfix will reject messages to non-exsisting recipients during the SMTP session with the client that attempts to deliver the message. See ADDRESS_VERIFICATION_README for details on implementing the reject_unverified_recipient option. There are two ways to prevent putting such load on the mailman LMTP server. First specify each valid recipient address in a transport table as shown in the initial example. Alternatively use the `reject_unverified_recipient` option and let the Postfix `verify`(8) daemon find out if the recipient address exists. This way Postfix will reject messages to non-exsisting recipients during the SMTP session with the client that attempts to deliver the message. See `ADDRESS_VERIFICATION_README` for details on implementing the reject_unverified_recipient option.
Line 125: Line 92:

{{{#!wiki warning

 
As of Mailman 3.0a4, the service name used in the Mailman generated<<BR>> `postfix_lmtp` file is not configurable. See [[https://bugs.launchpad.net/mailman/+bug/490030|bug 490030]].
}}}
Line 126: Line 100:
To create such a client add a new service (here: mailman) to the Postfix master.cf configuration file and add the configuration options which should override the default lmtp client behaviour:
To create such a client add a new service (here: ''mailman3'') to the Postfix `master.cf` configuration file and add the configuration options which should override the default lmtp client behaviour:
Line 129: Line 104:
 1. ==========================================================================
 1. service type  private unpriv  chroot  wakeup  maxproc command + args
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
Line 132: Line 107:

 1.
==========================================================================
# ==========================================================================
Line 135: Line 109:
mailman   unix  -       -       -       -       -       lmtp mailman3   unix  -       -       -       -       -       lmtp
Line 141: Line 115:
Then specify the new service in the transport table e.g. `/etc/postfix/mailman_lists:`

{{{
# key                           value
mailman@example.org             mailman3:inet:localhost
mailman-admin@example.org       mailman3:inet:localhost
mailman-bounces@example.org     mailman3:inet:localhost
mailman-confirm@example.org     mailman3:inet:localhost
...
}}}
Line 142: Line 127:

There are two variants of LMTP support in Exim transports.  The first of these exists as an option to the SMTP driver, instructing it to talk LMTP over the SMTP connection.  Such a transport will look like this: <<BR>>

{{{
mailman_remote_lmtp:
    driver = smtp
    protocol = lmtp
    hosts = localhost
    allow_localhost 
}}}

There is an independant LMTP transport driver, which is able to communicate using unix sockets.  For reference, this transport will look like the following, excluding any additional local configuration options:

{{{
mailman_local_lmtp:
    driver = lmtp
    socket = "/var/run/path/to/unix/socket"
    batch_max = 40
    user = mailman
}}}

<<BR>>

==== Email routing using Exim ====

The exim router should only need its transport option changed, the rest of the logic can remain the same.  One can consult the output of genaliases:

{{{
mailman_aliases:
    driver = accept
    domains = +local_domains
    condition = ${lookup${local_part}lsearch{/path/to/genaliases/output}}
    transport = mailman_local_lmtp
}}}

Alternatively, one can check directly for a list's existence:

{{{
 mailman_direct:
    driver = accept
    domains = +local_domains
    require_files = /mail/mailman/lists/${lc::$local_part}/config.pck
    local_part_suffix_optional
    local_part_suffix = -admin : \
                      -bounces : -bounces+* : \
                      -confirm : -confirm+* : \
                      -join : -leave : \
                      -owner : -request : \
                      -subscribe : -unsubscribe
    transport = mailman_local_lmtp

}}}

<<BR>> <<BR>>

==== Access Controls in Exim ====

Once you've set up routing and transports, you can use Exim's call forwards in an ACL (with the use_sender option) to determine whether the sender is permitted to post to the list. You should get a definitive answer, and this mechanism allows you to use a remote Mailman installation as if it were local - that is, you don't need to consult any local files. See section 40.40 of the Exim docs.

LMTP in Mailman 3

RFC 2033, Local Mail Transport Protocol (LMTP) provides Mailman with a unique opportunity to provide a better user experience when accepting initial postings. Two improvements in the process are available with LMTP.

First, we can eliminate most of the integration cruft we currently have with supporting multiple MTAs for incoming mail. Most of the major SMTP servers support LMTP delivery, so by providing an LMTP server in Mailman, the hope is that we can avoid all the crufty MTA-specific alias hackery.

Second, and perhaps more importantly, we can support better anti-backscatter and anti-spam defenses for messages sent to the mailing lists, by rejecting messages in the SMTP dialog instead of having to make the determination way later and sending a bounce message. Ian Eiloart describes what is possible.

How to set up LMTP delivery for Mailman 3

Here is detailed information about how Mailman should integrate with the various MTAs using LMTP.

Postfix

Ubuntu servers

If you're getting Connection refused errors, you might be hitting bug 340383. Try commenting out the 127.0.1.1 setting in your /etc/hosts file. Because Postfix runs under a chroot on Ubuntu, you'll also need to comment this out in /var/spool/postfix/etc/hosts.

Postfix uses the Postfix lmtp client to transport messages to a LMTP
server. You probably already have this in your Postfix master.cf file:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
...
lmtp      unix  -       -       -       -       -       lmtp
...

We're going to use a transport map to tell Postfix to deliver all messages
destined for a Mailman mailing list to its lmtp service. Fortunately, Mailman
is already able to write the correct transport map based on your configuration
settings. All you need to do is add a transport_maps setting in your
Postfix main.cf file:

transport_maps =
    ...
    hash:/path/to/mailman/var/data/postfix_lmtp
    ...

Now Postfix knows where it should transport messages to, but it doesn't know
yet it should accept messages for the given recipients - it doesn't know the
recipients are valid recipients. You can reuse the Mailman generated
transport map for this by adding the following to your main.cf file:

local_recipient_maps =
    ...
    hash:/path/to/mailman/var/data/postfix_lmtp
    ...

Virtual domain

If the mailman list addresses are part of $virtual_alias_domains or $virtual_mailbox_domains add postfix_lmtp to the listing of $virtual_alias_maps:

virtual_alias_maps =
    ...
    hash:/path/to/mailman/var/data/postfix_lmtp
    ...

Relay domain

If the mailman list addresses are part of the relay domain namespace add postfix_lmtp to the listing of $relay_recipient_maps:

relay_recipient_maps =
    ...
    hash:/path/to/mailman/var/data/postfix_lmtp
    ...

Once Postfix has been reloaded the new settings will take effect.

A dedicated list server

For a dedicated list server e.g. list.example.com simply add the hostname as key to a transport map and specify the mailman LMTP server as destination:

# key                           value
list.example.org                lmtp:inet:localhost

Be aware though that Postfix has no knowledge of valid recipients in the destination's (sub)domain (here: list.example.org) if you specify the list server as noted above. Postfix will accept and transport any message destined for list.example.org to the mailman LMTP server and it will be the LMTP servers task to reject messages for invalid or non-existing recipients.

There are two ways to prevent putting such load on the mailman LMTP server. First specify each valid recipient address in a transport table as shown in the initial example. Alternatively use the reject_unverified_recipient option and let the Postfix verify(8) daemon find out if the recipient address exists. This way Postfix will reject messages to non-exsisting recipients during the SMTP session with the client that attempts to deliver the message. See ADDRESS_VERIFICATION_README for details on implementing the reject_unverified_recipient option.

A dedicated lmtp client

As of Mailman 3.0a4, the service name used in the Mailman generated
postfix_lmtp file is not configurable. See bug 490030.

There may be situations where a dedicated lmtp client, that differs in its configuration from the default lmtp client settings, is required.

To create such a client add a new service (here: mailman3) to the Postfix master.cf configuration file and add the configuration options which should override the default lmtp client behaviour:

# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (yes)   (never) (100)
# ==========================================================================
...
mailman3   unix  -       -       -       -       -       lmtp
    -o lmtp_send_xforward_command=yes
    -o disable_dns_lookups=yes
...

Then specify the new service in the transport table e.g. /etc/postfix/mailman_lists:

# key                           value
mailman@example.org             mailman3:inet:localhost
mailman-admin@example.org       mailman3:inet:localhost
mailman-bounces@example.org     mailman3:inet:localhost
mailman-confirm@example.org     mailman3:inet:localhost
...

Exim

There are two variants of LMTP support in Exim transports.  The first of these exists as an option to the SMTP driver, instructing it to talk LMTP over the SMTP connection.  Such a transport will look like this:

mailman_remote_lmtp:
    driver = smtp
    protocol = lmtp
    hosts = localhost
    allow_localhost 

There is an independant LMTP transport driver, which is able to communicate using unix sockets.  For reference, this transport will look like the following, excluding any additional local configuration options:

mailman_local_lmtp:
    driver = lmtp
    socket = "/var/run/path/to/unix/socket"
    batch_max = 40
    user = mailman


Email routing using Exim

The exim router should only need its transport option changed, the rest of the logic can remain the same.  One can consult the output of genaliases:

mailman_aliases:
    driver = accept
    domains = +local_domains
    condition = ${lookup${local_part}lsearch{/path/to/genaliases/output}}
    transport = mailman_local_lmtp

Alternatively, one can check directly for a list's existence:

 mailman_direct:
    driver = accept
    domains = +local_domains
    require_files = /mail/mailman/lists/${lc::$local_part}/config.pck
    local_part_suffix_optional
    local_part_suffix =  -admin     : \
                      -bounces   : -bounces+* : \
                      -confirm   : -confirm+* : \
                      -join      : -leave     : \
                      -owner     : -request   : \
                      -subscribe : -unsubscribe
    transport = mailman_local_lmtp



Access Controls in Exim

Once you've set up routing and transports, you can use Exim's call forwards in an ACL (with the use_sender option) to determine whether the sender is permitted to post to the list. You should get a definitive answer, and this mechanism allows you to use a remote Mailman installation as if it were local - that is, you don't need to consult any local files. See section 40.40 of the Exim docs.

Sendmail


Comments

Ian Eiloart

William Mead has been working on LMTP code for me. He's produced implementations for version 3.0 and for version 2.2, with these tests applied after RCPT TO in the LMTP conversation:

1. message will be rejected if the list name is not known.
2. message will be accepted if the sender matches "accept_these_nonmembers".
3. message will be accepted if "generic_nonmember_action" is not reject.
4. message will be accepted if the sender is a list member.
5. if we get this far, the message will be rejected - the sender is a non-member of a closed list.

We could also reject other members if they're moderated, for example. However, we've adopted the view that it is relatively safe to generate a bounce message for someone who is a member of the list.

William has completely reimplemented the SMTPD code in Python, to support ENHANCEDSTATUSCODES, because the LMTP RFC requires that - even though the examples in the RFC don't show them being used! However, the code doesn't implement PIPELINING - also required by LMTP - because the underlying ASYNCHAT/ASYNCORE architecture doesn't seem to support it. We discovered that advertising PIPELINING causes the test smtp client to fail, but we've not even thought about how to fix that - LMTP clients which are re-implementations of SMTP clients might just live with the fact that PIPELINING isn't advertised.

William's code is at https://code.launchpad.net/~wilunix

The LMTP queue runner allows us to run Mailman on a server that's unrelated to the main MTA. With an Exim MTA, you could use a recipient callout to verify that the sender is permitted to post to the list, before accepting the message for deliver. This means that rejecting an unwanted message should not create collateral spam.

MailmanWiki: DEV/LMTP process (last edited 2009-11-29 10:51:39 by barry)