Differences between revisions 1 and 8 (spanning 7 versions)
Revision 1 as of 2016-07-03 20:20:41
Size: 22786
Editor: msapiro
Comment:
Revision 8 as of 2017-01-06 04:08:54
Size: 11459
Editor: msapiro
Comment: Correct a path /opt/git -> /opt/mailman/git
Deletions are marked like this. Additions are marked like this.
Line 1: Line 1:
=== Introduction ===
Line 10: Line 11:
{{{#!wiki note
Postorius and HyperKitty have now changed from Mozilla Persona supplemented with Python Social Auth to Django Allauth for authentication, and the templating in Mailman Core has changed. The changes associated with this are outlined towards the end of this document.
}}}
Line 12: Line 16:
=== lists.mailman3.org ===
On lm3o, I created /opt/mailman and cloned mailman-bundler into /opt/mailman/mailman-bundler. I more or less followed its docs for a production install, but I decided not to use virtualenvs, so I set buildout.cfg for 'production' and ran buildout not in a virtualenv - also I probably had to ensure I had up to date setuptools and pip.
Line 13: Line 19:
On lm3o, I cloned mailman-bundler into /opt/mailman/mailman-bundler. I more or less followed its docs for a production install, but I decided not to use virtualenvs, so I set buildout.cfg for 'production' and ran buildout not in a virtualenv - also I probably had to ensure I had up to date setuptools and pip. I installed ruby-sass and postgreSQL. I installed psycopg2 with both pip and pip3. I also installed gunicorn as the WSGI server for nginx and [[attachment:lm3o_nginx.txt|configured]] ngnix. I configured gunicorn as an [[attachment:lm3o_gunicorn.txt|upstart job]] to run as user:group mailman:mailman and set its [[attachment:lm3o_gunicorn.conf.txt|config]] as recommended.
Line 15: Line 21:
I installed ruby-sass and postgreSQL. I installed psycopg2 with both pip and pip3. I also installed gunicorn as the WSGI server for nginx. The nginx config is [1]. I configured gunicorn as an upstart job to run as user:group mailman:mailman [2] and set its config as recommended [3]. I created the mailmanweb postgreSQL DB and a postgreSQL mailman user.
Line 19: Line 25:
To this end, I cloned the hyperkitty, mailmanclient, postorius, mailman and mailman-hyperkitty branches into /opt/git. and ran their respective 'setup.py install' scripts with the appropriate system Python (3.4 for mailman and mailman-hyperkitty and 2.7 for the rest). The next step is the hardest. for those packages in mailman-bundler/eggs that had corresponding system package (probably in /usr/lib/python2.7/dist-packages) that could be imported in the system Python, I just deleted the mailman-bundler/eggs/ package and similarly for those Python 3 packages in mailman-bundler/venv-3.4/lib/python3.4/site-packeges that had a corresponding system package. To this end, I cloned the hyperkitty, mailmanclient, postorius, mailman and mailman-hyperkitty branches into /opt/mailman/git. and ran their respective 'setup.py install' scripts with the appropriate system Python (3.4 for mailman and mailman-hyperkitty and 2.7 for the rest). The next step is the hardest. For those packages in mailman-bundler/eggs that had corresponding system package (probably in /usr/lib/python2.7/dist-packages) that could be imported in the system Python, I just deleted the mailman-bundler/eggs/ package and similarly for those Python 3 packages in mailman-bundler/venv-3.4/lib/python3.4/site-packeges that had a corresponding system package.
Line 23: Line 29:
There were some issues because I used only Django 1.9. django-haystack 2.4 is not compatible with Django 1.9, so I cloned the 2.5.dev branch from github and installed that. There were some issues because I used only Django 1.9. django-haystack 2.4 is not compatible with Django 1.9, so I cloned the django-haystack 2.5.dev branch from github and installed that.
Line 27: Line 33:
The remainder consisted of putting desired configuration stuff in mailman-bundler/mailman_web/settings_local.py [4], and editing mailman-bundler/deployment/mailman.cfg [5], mailman-bundler/deployment/mailman-hyperkitty.cfg [6] and mailman-bundler/mailman_web/urls.py [7]. The remainder consisted of putting desired configuration in [[attachment:lm3o_settings_local.txt|mailman-bundler/mailman_web/settings_local.py]], and editing [[attachment:lm3o_mailman.cfg.txt|mailman-bundler/deployment/mailman.cfg]], [[attachment:lm3o_hyperkitty.cfg.txt|mailman-bundler/deployment/mailman-hyperkitty.cfg]] and [[attachment:lm3o_urls.txt|mailman-bundler/mailman_web/urls.py]].
Line 29: Line 35:
I also tried to create an upstart script for mailman, but despite trying all options, I couldn't get it to get the proper demonized PID so I went with an init.d script [8]. I also tried to create an upstart script for mailman, but despite trying all options, I couldn't get it to get the proper demonized PID so I went with an [[attachment:mailman_init.txt|init.d script]].
Line 31: Line 37:
The Mailman specific Postfix things are very straightforward [9]. The Mailman specific Postfix [[attachment:lm3o_postfix.txt|things]] are very straightforward.
Line 33: Line 39:
For authentication, I couldn't get Google OpenID to work, but I was able to use OAuth2. I tried Twitter, but you can't get email address from Twitter so thats out. See [4] for details. Also, I added a bit to mailman-bundler/bin/mailman-post-update to do For authentication, I couldn't get Google OpenID to work, but I was able to use OAuth2. I tried Twitter, but you can't get email address from Twitter so that's out. See [[attachment:lm3o_settings_local.txt|settings_local.py]] for details. Also, I added a bit to mailman-bundler/bin/mailman-post-update to do
{{{
Line 38: Line 44:
}}}
Line 41: Line 47:
Because there are

There are several default templates that have incorrect or generic URLs for the web UI. To make these better, I installed these templates in /opt/mailman/mailman-bundler/var/templates/site/en/:
 * [[attachment:footer-generic.txt]]
 * [[attachment:masthead.txt]]
 * [[attachment:newlist.txt]]
 * [[attachment:probe.txt]]
 * [[attachment:welcome.txt]]
and for each supported domain I installed in /opt/mailman/mailman-bundler/templates/domains/EMAIL_DOMAIN/en/:
 * [[attachment:postack.txt]]
with the literal `EMAIL_DOMAIN` replaced by the actual domain name.
=== mail.python.org ===
Line 60: Line 73:
Install sass. Install ruby-sass.
Line 62: Line 75:
Install apache mod_wsgi. The Apache config is [10] Install apache mod_wsgi. The Apache wsgi config is [[attachment:mpo_wsgi.txt]].
Line 64: Line 77:
Make mailman_web/local_settings.py [4] with differences [11] Make [[attachment:lm3o_settings_local.txt|mailman_web/local_settings.py]] with [[attachment:mpo_settings_diff.txt|differences]]
Line 66: Line 79:
Edit mailman_web/urls.py (change urlpatterns to list for django 1.9 and remove {"SSL": True\} [7] Edit [[attachment:lm3o_urls.txt|mailman_web/urls.py]] (change urlpatterns to list for django 1.9 and remove `{"SSL": True\}`
Line 68: Line 81:
Edit base_url and api_key in /opt/mailman/mailman-bundler/deployment/mailman-hyperkitty.cfg [6] (values are different) Edit base_url and api_key in [[attachment:lm3o_hyperkitty.cfg.txt|/opt/mailman/mailman-bundler/deployment/mailman-hyperkitty.cfg]] (values are different).
Line 84: Line 97:
Add custom templates as above for lm3o.
=== Postfix issues ===
Line 88: Line 103:
Use virtual_alias_maps to map list(-*)@python.org to list(-*)@x.python.org, and then use transport_maps to relay list(-*)@x.python.org to Mailman's LMTP runner in the usual way, and finally modify Mailman's LMTP runner to rewrite the x.python.org recipient domain to python.org so the address corresponds to what Mailman expects. Use virtual_alias_maps to map `list(-*)@python.org` to `list(-*)@x.python.org`, and then use transport_maps to relay `list(-*)@x.python.org` to Mailman's LMTP runner in the usual way, and finally modify Mailman's LMTP runner to rewrite the `x.python.org` recipient domain to `python.org` so the address corresponds to what Mailman expects.
Line 90: Line 105:
The details involve several pieces. Add a section [12] to mailman-bundler/deployment/mailman.cfg to reference mailman-bundler/deployment/postfix.cfg [13] which in turn sets postmap_command to mailman-bundler/bin/postmap_command [14] which creates additional files. Reference these files in Postfix [15] and patch mailman/runners/lmtp.py [16].




[1] This is the nginx config for Mailman. The configs for http and https are essentially the same except for the added SSL stuff for https. The critical stuff is the /static/ location and the proxy stuff for proxying to gunicorn at port 8000.

server {
 listen 80 default_server;
 # listen [::]:80 default_server ipv6only=on;

 root /usr/share/nginx/html;
 index index.html index.htm;

 server_name lists.mailman3.org;

 location / {
  proxy_pass http://127.0.0.1:8000/;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_redirect off;
 }

 location /static/ {
  alias /var/spool/mailman-web/static/;
 }

 location /robots.txt {
  alias /var/spool/mailman-web/static/hyperkitty/robots.txt;
 }

 location /favicon.ico {
  alias /var/www/listorg/images/favicon.ico;
 }

}

server {
 listen 443 default_server;
 # listen [::]:443 default_server ipv6only=on;

 root /usr/share/nginx/html;
 index index.html index.htm;

 server_name lists.mailman3.org;

 ssl on;
 ssl_certificate /etc/letsencrypt/live/mirror.mailman3.org/fullchain.pem;
 ssl_certificate_key /etc/letsencrypt/live/mirror.mailman3.org/privkey.pem;

 ssl_session_timeout 5m;

 ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2;
 ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES";
 ssl_prefer_server_ciphers on;

 location / {
  proxy_pass http://127.0.0.1:8000/;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Host $host;
  proxy_set_header X-Forwarded-Server $host;
  proxy_set_header X-Forwarded-Proto $scheme;
  proxy_redirect off;
 }

 location /static/ {
  alias /var/spool/mailman-web/static/;
 }

 location /robots.txt {
  alias /var/spool/mailman-web/static/hyperkitty/robots.txt;
 }

 location /favicon.ico {
  alias /var/www/listorg/images/favicon.ico;
 }

}

[2] gunicorn upstart (/etc/init/gunicorn.conf)

description "gunicorn for Mailman"

start on runlevel [2345]
stop on runlevel [016]

respawn
respawn limit 10 5

setuid mailman
setgid mailman

chdir /opt/mailman/mailman-bundler

exec /opt/mailman/mailman-bundler/bin/gunicorn -c /opt/mailman/mailman-bundler/deployment/gunicorn.conf mailman_web.wsgi:application

{3] gunicorn config (/opt/mailman/mailman-bundler/deployment/gunicorn.conf)

bind = ['127.0.0.1:8000']
proc_name = "mailman-web"
chdir = "/opt/mailman/mailman-bundler"
pidfile = "/opt/mailman/mailman-bundler/var/gunicorn.pid"
accesslog = "/opt/mailman/mailman-bundler/var/logs/access.log"
errorlog = "/opt/mailman/mailman-bundler/var/logs/error.log"
#daemon = True

[4] mailman-bundler/mailman_web/settings_local.py (not the real keys/secrets)

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django secret'

MAILMAN_ARCHIVER_KEY = 'arch_secret'

ADMINS = (
     ('Mailman Admin', 'mailman@localhost'),
)

# Hosts/domain names that are valid for this site; required if DEBUG is False
# See https://docs.djangoproject.com/en/1.9/ref/settings/#allowed-hosts
#ALLOWED_HOSTS = ['*']
ALLOWED_HOSTS = ['localhost',
                 '127.0.0.1',
                 'lists.mailman3.org',
                 'mail.mailman3.org',
                 'mailman.iad1.psf.io',
                 'mail.falconframework.org',
                 '104.239.228.201',
                ]

# And for BrowserID too, see
# http://django-browserid.rtfd.org/page/user/settings.html#django.conf.settings.BROWSERID_AUDIENCES
BROWSERID_AUDIENCES = [ "http://localhost",
                        "http://localhost:8000",
                        "http://127.0.0.1:8000",
                        "http://lists.mailman3.org",
                        "http://mail.falconframework.org",
                        "https://localhost",
                        "https://localhost:8000",
                        "https://127.0.0.1:8000",
                        "https://lists.mailman3.org",
                        "https://mail.falconframework.org",
                      ]


## Email confirmation / address activation
# Add a from-address for email-confirmation:
# EMAIL_CONFIRMATION_FROM = 'postmaster@example.org'
EMAIL_CONFIRMATION_FROM = 'mailman@mailman3.org'

MAILMAN_ARCHIVER_FROM = ('127.0.0.1',
                         '::1',
                         '::ffff:127.0.0.1',
                         '104.239.228.201',
                        )

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2', # Last part is one of 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'.
        'NAME': 'mailmanweb', # Example, change as needed
        'USER': 'mailman',
        'PASSWORD': 'db_passwd',
        'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP.
        'PORT': '5432', # Set to empty string for default.
    }
}

# If you're behind a proxy, use the X-Forwarded-Host header
# See https://docs.djangoproject.com/en/1.8/ref/settings/#use-x-forwarded-host
USE_X_FORWARDED_HOST = True
# And if your proxy does your SSL encoding for you, set SECURE_PROXY_SSL_HEADER
# see https://docs.djangoproject.com/en/1.5/ref/settings/#secure-proxy-ssl-header
SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

# Use SSL when logged in. You need to enable the SSLRedirect middleware for
# this feature to work.
USE_SSL = True

TIME_ZONE = 'UTC'

AUTHENTICATION_BACKENDS = (
    #'social.backends.open_id.OpenIdAuth',
    # http://python-social-auth.readthedocs.org/en/latest/backends/google.html
    #'social.backends.google.GoogleOpenId',
    'social.backends.google.GoogleOAuth2',
    # Twitter isn't working ...
    # Rumor has it you can't get email from twitter :(
    #'social.backends.twitter.TwitterOAuth',
    # Yahoo works, but don't encourage Yahoo until we get DMARC mitigations.
    #'social.backends.yahoo.YahooOpenId',
    'social.backends.facebook.FacebookOAuth2',
    'django_browserid.auth.BrowserIDBackend',
    'django.contrib.auth.backends.ModelBackend',
)

SOCIAL_AUTH_TWITTER_KEY = 'twitter key'
SOCIAL_AUTH_TWITTER_SECRET = 'twitter secret'
SOCIAL_AUTH_TWITTER_SCOPE = ['email', 'name']

SOCIAL_AUTH_GOOGLE_OAUTH2_KEY = 'google key'
SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET = 'google secret'
SOCIAL_AUTH_GOOGLE_OAUTH2_USE_DEPRECATED_API = True

SOCIAL_AUTH_FACEBOOK_KEY = 'fb key'
SOCIAL_AUTH_FACEBOOK_SECRET = 'fb secret'
SOCIAL_AUTH_FACEBOOK_SCOPE = ['email', 'public_profile']
SOCIAL_AUTH_FACEBOOK_PROFILE_EXTRA_PARAMS = {
    'fields': 'name, email',
}


[5] mailman-bundler/deployment/mailman.cfg

changes are site_owner, [database] and [shell]

# This is the absolute bare minimum base configuration file. User supplied
# configurations are pushed onto this.

[mailman]
# This address is the "site owner" address. Certain messages which must be
# delivered to a human, but which can't be delivered to a list owner (e.g. a
# bounce from a list owner), will be sent to this address. It should point to
# a human.
site_owner: mailman@mailman3.org
layout: here

[paths.here]
# Everything in the same directory
var_dir: /opt/mailman/mailman-bundler/var

[database]
class: mailman.database.postgresql.PostgreSQLDatabase
#url: postgres://mailman:mailman-db-password@localhost/mailman
url: postgres://mailman:db_passwd@localhost/mailmanweb

[archiver.hyperkitty]
class: mailman_hyperkitty.Archiver
enable: yes
configuration: /opt/mailman/mailman-bundler/deployment/mailman-hyperkitty.cfg

[archiver.prototype]
enable: yes

[shell]
history_file: $var_dir/history.py

#[logging.database]
#level: debug

[6] mailman-bundler/deployment/mailman-hyperkitty.cfg

changes are base_url and api_key

# This is the mailman extension configuration file to enable HyperKitty as an
# archiver. Remember to add the following lines in the mailman.cfg file:
#
# [archiver.hyperkitty]
# class: mailman_hyperkitty.Archiver
# enable: yes
# configuration: /path/to/here/hyperkitty.cfg
#

[general]
base_url: https://lists.mailman3.org/archives
api_key: arch_secret

[7] mailman-bundler/mailman_web/urls.py

Changes are converting url patterns to a list for Django 1.9 and removing '{"SSL": True\} from a couple of URLs as it caused problems.

from django.conf.urls import include, url
from django.core.urlresolvers import reverse_lazy
from django.views.generic import RedirectView

# Comment the next two lines to disable the admin:
from django.contrib import admin
admin.autodiscover()

urlpatterns = [url(r'^$', RedirectView.as_view(url=reverse_lazy('hyperkitty.views.index.index'))),
               url(r'^mailman3/', include('postorius.urls')),
               url(r'^archives/', include('hyperkitty.urls')),
               url(r'', include('social.apps.django_app.urls', namespace='social')),
               url(r'', include('django_browserid.urls')),
              ]

[8] /etc/init.d/mailman

### BEGIN INIT INFO
# Provides: GNU Mailman
# Required-Start: $local_fs $network
# Required-Stop: $local_fs $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Mailman Service
# Description: service control for Mailman
### END INIT INFO

PATH=/opt/mailman/mailman-bundler/bin:/usr/sbin:/usr/bin:/bin:/sbin:
DESC="GNU Mailman service"
DAEMON=/opt/mailman/mailman-bundler/bin/mailman
NAME=mailman
USER=mailman
GROUP=mailman

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.2-14) to ensure that this file is present
# and status_of_proc is working.
. /lib/lsb/init-functions

case "$1" in
  start)
        [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
        sudo -u $USER $DAEMON start
        ;;
  stop)
        [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
        sudo -u $USER $DAEMON stop
        ;;
  status)
        sudo -u $USER $DAEMON status
        ;;
  reopen)
        sudo -u $USER $DAEMON reopen
        ;;
  restart)
        log_daemon_msg "Restarting $DESC" "$NAME"
        sudo -u $USER $DAEMON restart
        ;;
  *)
        echo "Usage: $SCRIPTNAME {start|stop|status|reopen|restart}" >&2
        exit 3
        ;;
esac


[9] MM 3 specific Postfix additions lm3o

local_recipient_maps = hash:/opt/mailman/mailman-bundler/var/data/postfix_lmtp
    proxy:unix:passwd.byname $alias_maps
relay_domains = hash:/opt/mailman/mailman-bundler/var/data/postfix_domains
transport_maps = hash:/opt/mailman/mailman-bundler/var/data/postfix_lmtp

[10] Apache mod_wsgi config.

This goes in a global section.

WSGIDaemonProcess mailman-web display-name=mailman-web maximum-requests=1000 umask=0002 user=mailman group=mailman python-path=/opt/mailman/mailman-bundler/venv/lib/python2.7/site-packages:/opt/mailman/mailman-bundler/eggs home=/opt/mailman/mailman-bundler/var
WSGIRestrictSignal Off

(I'm not sure that WSGIRestrictSignal Off is required, but it was in the provided example so I kept it. I also made changes to WSGIDaemonProcess based on my own mod_wsgi experience elsewhere.)

This goes in the VirtualHost block for the domain.

        # Mailman 3 stuff
        Alias /static "/var/spool/mailman-web/static"
        <Directory "/var/spool/mailman-web/static">
            Require all granted
        </Directory>
        WSGIScriptAlias /mm3 /opt/mailman/mailman-bundler/bin/mailman-web.wsgi
        <Directory "/opt/mailman/mailman-bundler/bin">
            <Files mailman-web.wsgi>
                Order deny,allow
                Allow from all
                Require all granted
            </Files>
            WSGIProcessGroup mailman-web
        </Directory>


[11] Differences between lm3o settings_local.py [4] and mpo.

ALLOWED_HOSTS = ['localhost',
                 '127.0.0.1',
                 'mail.python.org'
                ]

BROWSERID_AUDIENCES = [ "http://localhost",
                        "http://mail.python.org",
                        "https://localhost",
                        "https://mail.python.org",
                      ]

EMAIL_CONFIRMATION_FROM = 'mailman@python.org'

MAILMAN_ARCHIVER_FROM = ('127.0.0.1',
                         '::1',
                         '::ffff:127.0.0.1',
                         '188.166.95.178',
                        )


[12] Addition to mailman.cfg for Postfix kludge

[mta]
configuration: /opt/mailman/mailman-bundler/deployment/postfix.cfg


[13] Content of postfix.cfg

[postfix]
# Additional configuration variables for the postfix MTA.

# This variable describe the program to use for regenerating the transport map
# db file, from the associated plain text files. The file being updated will
# be appended to this string (with a separating space), so it must be
# appropriate for os.system().
postmap_command: /opt/mailman/mailman-bundler/bin/postmap_command

[14] Content of postmap_command script.

#!/bin/bash

umask 0002
LMTP=/opt/mailman/mailman-bundler/var/data/postfix_lmtp
XLMTP=/opt/mailman/mailman-bundler/var/data/postfix_xlmtp
VMAP=/opt/mailman/mailman-bundler/var/data/postfix_vmap
DOM=/opt/mailman/mailman-bundler/var/data/postfix_domains
XDOM=/opt/mailman/mailman-bundler/var/data/postfix_xdomains

if [[ $1 =~ postfix_lmtp$ ]] ; then
    sed < $LMTP > $XLMTP -e 's/@python.org/@x.python.org/'
    sed < $LMTP > $VMAP -r -e 's/(^[^ ]*)@python.org( *).*/\1@python.org\2\1@x.python.org/'
    /usr/sbin/postmap $XLMTP
    /usr/sbin/postmap $VMAP
elif [[ $1 =~ postfix_domains$ ]] ; then
    sed < $DOM > $XDOM -e 's/^python.org/x.python.org/'
    /usr/sbin/postmap $XDOM
else
    /usr/sbin/postmap $1
fi


[15] Postfix main.cf additions for mpo.

relay_domains = hash:/opt/mailman/mailman-bundler/var/data/postfix_xdomains
transport_maps = cdb:/etc/postfix/transport hash:/opt/mailman/mailman-bundler/var/data/postfix_xlmtp
virtual_alias_maps = cdb:/etc/postfix/virtual-python cdb:/etc/postfix/virtual-snakebite cdb:/etc/postfix/virtual-usergroups cdb:/etc/postfix/virtual-raspberry hash:/srv/mailman/data/virtual-mailman hash:/opt/mailman/mailman-bundler/var/data/postfix_vmap

Note that the order of hash:/srv/mailman/data/virtual-mailman (MM2 maps) and hash:/opt/mailman/mailman-bundler/var/data/postfix_vmap (MM3 maps) determines in the case of MM2 and MM3 lists with the same name, which list gets the mail.


[16] lmtp/py patch

--- git/mailman/src/mailman/runners/lmtp.py 2016-06-05 16:58:06.255217017 -0400
+++ mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman-3.1.0-py3.4.egg/mailman/runners/lmtp.py 2016-06-21 00:47:29.497817768 -0400
@@ -118,6 +118,9 @@
         subaddress may be None if this is the list's posting address.
     """
     localpart, domain = address.split('@', 1)
+ # Kludge for x.python.org <-> python.org
+ if domain == 'x.python.org':
+ domain = 'python.org'
     localpart = localpart.split(config.mta.verp_delimiter, 1)[0]
     parts = localpart.split(DASH)
     if parts[-1] in SUBADDRESS_NAMES:
The details involve several pieces. Add a [[attachment:mpo_mailman.cfg.add.txt|section]] to mailman-bundler/deployment/mailman.cfg to reference [[attachment:mpo_postfix.cfg.txt|mailman-bundler/deployment/postfix.cfg]] which in turn sets postmap_command to [[attachment:mpo_postmap.txt|mailman-bundler/bin/postmap_command]] which creates additional files. Reference these files in [[attachment:mpo_postfix_main.txt|Postfix]] and [[attachment:mpo_lmtp_patch.txt|patch]] mailman/runners/lmtp.py.
== Recent changes ==
=== Postfix ===
At the current time, there is are proposals to modify the [[https://gitlab.com/mailman/mailman/merge_requests/202|core]] and [[https://gitlab.com/mailman/mailmanclient/merge_requests/18|mailman client]] to add an alias_domain attribute to the domain object to deal with the above Postfix configuration issue. Ultimately Postorius will also need to be updated to set/display the alias_domain.
=== Django Haystack ===
django-haystack 2.5 has been released and should be installed by bundler so it shouldn't be necessary to do anything special for Django Haystack compatibility with Django 1.9.
=== Templating Changes ===
The core's templating has changed. The information above about custom templates is no longer valid. The custom templates I am using are now all in /opt/mailman/mailman-bundler/var/templates/site/en/ as follows:
 * [[attachment:domain:admin:notice:new-list.txt]]
 * [[attachment:list:admin:action:post.txt]]
 * [[attachment:list:member:digest:masthead.txt]]
 * [[attachment:list:member:generic:footer.txt]]
 * [[attachment:list:user:notice:post.txt]]
 * [[attachment:list:user:notice:probe.txt]]
 * [[attachment:list:user:notice:welcome.txt]]
These are the lm3o templates. The mpo templates are the same except for URL.
=== Django Allauth Changes ===
The change to Django Allauth as well as other changes (see below) required changes to files in /opt/mailman/mailman-bundler/mailman_web/. The three changed files are:
 * [[attachment:urls.py]]
 * [[attachment:production.py]]
 * [[attachment:settings_local.py]]
These are the lm3o versions of these files with some passwords/keys elided. The differences for mpo are the same as for the prior versions. It is not clear that there aren't things in settings_local.py that should really be in production.py, but this is the config that works for me with social auth for Google, Facebook, !GitHub and !GitLab. Note that the keys and secrets for social auth providers are now set in the database via the Django admin web UI.
=== Other Changes ===
Parts of Postorius and HyperKitty have been split out into a [[https://gitlab.com/mailman/django-mailman3|django-mailman3]] branch which needs to be cloned and installed. It is installed along with Postorius and HyperKitty in the same way as they are installed. This also requires 'django_mailman3' to be added to INSTALLED_APPS in the Django config. See above.

Introduction

I have now installed Mailman 3 for production on two different servers. This is intended to be documentation of my decisions and experiences in doing so. It will be more of a narrative than a how-to, but I hope a how-to can be distilled from it.

The two servers are lists.mailman3.org (lm3o) and mail.python.org (mpo). Pre-existing things on those servers forced some decisions which were different between them, but the actual MM 3 installation was pretty much up to me, yet I did it differently the second time due to things I thought I'd learned the first time.

A fundamental step in any installation is to ensure that you have the necessary infrastructure for reliably sending mail. This mostly involves DNS records. I won't go into detail, but you need an A and maybe AAAA record for your MAIL FROM domain and rDNS PTR records for the IPs pointing back to the domain. You may also need an MX record for your email domain pointing to the Mailman server. Also good is a mechanism for DKIM signing outgoing mail and DKIM and SPF records for the domain.

This already existed on mpo, but needed to be set up on lm3o.

lm3o is the same server as mirror.list.org/mirror.mailman3.org. This is a mirror of the GNU Mailman web site and was already set up using nginx as a web server. Thus, the MM 3 installation here uses nginx as the web server proxying to gunicorn for WSGI support.

Postorius and HyperKitty have now changed from Mozilla Persona supplemented with Python Social Auth to Django Allauth for authentication, and the templating in Mailman Core has changed. The changes associated with this are outlined towards the end of this document.

For both installs I started with mailman-bundler.

lists.mailman3.org

On lm3o, I created /opt/mailman and cloned mailman-bundler into /opt/mailman/mailman-bundler. I more or less followed its docs for a production install, but I decided not to use virtualenvs, so I set buildout.cfg for 'production' and ran buildout not in a virtualenv - also I probably had to ensure I had up to date setuptools and pip.

I installed ruby-sass and postgreSQL. I installed psycopg2 with both pip and pip3. I also installed gunicorn as the WSGI server for nginx and configured ngnix. I configured gunicorn as an upstart job to run as user:group mailman:mailman and set its config as recommended.

I created the mailmanweb postgreSQL DB and a postgreSQL mailman user.

The main departure from a bundler production install is that I wanted to install the various Mailman components (and dependencies) system wide from the heads of the respective gitlab branches.

To this end, I cloned the hyperkitty, mailmanclient, postorius, mailman and mailman-hyperkitty branches into /opt/mailman/git. and ran their respective 'setup.py install' scripts with the appropriate system Python (3.4 for mailman and mailman-hyperkitty and 2.7 for the rest). The next step is the hardest. For those packages in mailman-bundler/eggs that had corresponding system package (probably in /usr/lib/python2.7/dist-packages) that could be imported in the system Python, I just deleted the mailman-bundler/eggs/ package and similarly for those Python 3 packages in mailman-bundler/venv-3.4/lib/python3.4/site-packeges that had a corresponding system package.

I also went through the mailman-bundler/bin/* scripts and removed now non-existent paths from the list added to sys.path.

There were some issues because I used only Django 1.9. django-haystack 2.4 is not compatible with Django 1.9, so I cloned the django-haystack 2.5.dev branch from github and installed that.

There are also a lot of deprecation warnings in logs, some of which I've tried to address.

The remainder consisted of putting desired configuration in mailman-bundler/mailman_web/settings_local.py, and editing mailman-bundler/deployment/mailman.cfg, mailman-bundler/deployment/mailman-hyperkitty.cfg and mailman-bundler/mailman_web/urls.py.

I also tried to create an upstart script for mailman, but despite trying all options, I couldn't get it to get the proper demonized PID so I went with an init.d script.

The Mailman specific Postfix things are very straightforward.

For authentication, I couldn't get Google OpenID to work, but I was able to use OAuth2. I tried Twitter, but you can't get email address from Twitter so that's out. See settings_local.py for details. Also, I added a bit to mailman-bundler/bin/mailman-post-update to do

    cd $STATIC_DIR/hyperkitty/img/login
    ln -s google.png google-oauth2.png
    cd $OLDPWD

for the login graphic.

There are several default templates that have incorrect or generic URLs for the web UI. To make these better, I installed these templates in /opt/mailman/mailman-bundler/var/templates/site/en/:

and for each supported domain I installed in /opt/mailman/mailman-bundler/templates/domains/EMAIL_DOMAIN/en/:

with the literal EMAIL_DOMAIN replaced by the actual domain name.

mail.python.org

The installation on mail.python.org was similar in many ways but different in others. The major differences are because mpo is an established server running Apache and Postfix and supporting over 300 Mailman 2.1 lists in the python.org domain which is a Postfix virtual alias domain. This complicated the Postfix configuration because we wanted Mailman 3 lists to also be in the python.org email domain and Postfix virtual alias domains don't honor transport maps which is the normal way of delivering mail from Postfix to MM 3.

Another difference is I decided to use virtualenvs for this install.

Much of the install was the same. I have some notes of what was done, and I will only elaborate differences.

Steps were:

Create /opt/mailman owned by mailman.

As the mailmanuser, git clone mailman-bundler.

Basically follow bundler docs.

Run buildout as production.

Install ruby-sass.

Install apache mod_wsgi. The Apache wsgi config is mpo_wsgi.txt.

Make mailman_web/local_settings.py with differences

Edit mailman_web/urls.py (change urlpatterns to list for django 1.9 and remove {"SSL": True\}

Edit base_url and api_key in /opt/mailman/mailman-bundler/deployment/mailman-hyperkitty.cfg (values are different).

Install postgresql create mailmanweb DB and mailman user.

Install postgresql-server-dev-9.3 and then pip install psycopg2 in both venvs.

Make /opt/mailman/git/ and clone django-haystack mailman mailman-hyperkitty hyperkitty mailmanclient postorius repos and run the setups in the appropriate venv.

Remove older mailman and mailman-hyperkitty from venv-3.4/lib/...

In eggs/ for everything with a version in venv/lib/..., remove the eggs/ subdir and replace it with a symlink to the venv/lib/... version. Link both Django 1.8 and 1.9 to the Django 1.9 in venv/lib/...

Create django superuser.

Update Oauth providers to recognize the additional callback domain.

Add custom templates as above for lm3o.

Postfix issues

A major issue on mpo was trying to have MM 3 lists have @python.org email addresses so these wouldn't need to change upon list migration. In cases where the list domain is in Postfix mydestinations, this is straightforward, but when the list domain is a virtual alias domain, it's more difficult. What I did is a bit of a kludge. We plan to make this a feature so it too will be straightforward, but it isn't there yet.

The solution is as follows:

Use virtual_alias_maps to map list(-*)@python.org to list(-*)@x.python.org, and then use transport_maps to relay list(-*)@x.python.org to Mailman's LMTP runner in the usual way, and finally modify Mailman's LMTP runner to rewrite the x.python.org recipient domain to python.org so the address corresponds to what Mailman expects.

The details involve several pieces. Add a section to mailman-bundler/deployment/mailman.cfg to reference mailman-bundler/deployment/postfix.cfg which in turn sets postmap_command to mailman-bundler/bin/postmap_command which creates additional files. Reference these files in Postfix and patch mailman/runners/lmtp.py.

Recent changes

Postfix

At the current time, there is are proposals to modify the core and mailman client to add an alias_domain attribute to the domain object to deal with the above Postfix configuration issue. Ultimately Postorius will also need to be updated to set/display the alias_domain.

Django Haystack

django-haystack 2.5 has been released and should be installed by bundler so it shouldn't be necessary to do anything special for Django Haystack compatibility with Django 1.9.

Templating Changes

The core's templating has changed. The information above about custom templates is no longer valid. The custom templates I am using are now all in /opt/mailman/mailman-bundler/var/templates/site/en/ as follows:

These are the lm3o templates. The mpo templates are the same except for URL.

Django Allauth Changes

The change to Django Allauth as well as other changes (see below) required changes to files in /opt/mailman/mailman-bundler/mailman_web/. The three changed files are:

These are the lm3o versions of these files with some passwords/keys elided. The differences for mpo are the same as for the prior versions. It is not clear that there aren't things in settings_local.py that should really be in production.py, but this is the config that works for me with social auth for Google, Facebook, GitHub and GitLab. Note that the keys and secrets for social auth providers are now set in the database via the Django admin web UI.

Other Changes

Parts of Postorius and HyperKitty have been split out into a django-mailman3 branch which needs to be cloned and installed. It is installed along with Postorius and HyperKitty in the same way as they are installed. This also requires 'django_mailman3' to be added to INSTALLED_APPS in the Django config. See above.

MailmanWiki: DOC/Mailman 3 installation experience (last edited 2021-10-15 19:24:52 by msapiro)