|
Size: 22786
Comment:
|
Size: 9041
Comment: Added a comment section.
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| === Introduction === | |
| Line 11: | Line 12: |
| For both installs I started with mailman-bundler. | === Update === I have done a significant reinstallation on both servers. <<Action(recall,The prior version of this page,rev=10)>> is no longer applicable to these servers. I have also installed MM 3 on a third server. I have now evolved my process and installations to a stable point. It is this configuration that I will describe. {{{#!wiki note The information about templates in the lists.mailman3.org section in <<Action(recall,the prior version of this page,rev=10)>> is still valid, but the path /opt/mailman/mailman-bundler/var/templates/... becomes /opt/mailman/mm/var/templates/... }}} === Preliminaries === Both lm3o and mpo already had sass and memcached installed via |
| Line 13: | Line 23: |
| On lm3o, I cloned mailman-bundler into /opt/mailman/mailman-bundler. I more or less followed its docs for a production install, but I decided not to use virtualenvs, so I set buildout.cfg for 'production' and ran buildout not in a virtualenv - also I probably had to ensure I had up to date setuptools and pip. | `apt install ruby-sass`<<BR>> `apt install memcached` |
| Line 15: | Line 26: |
| I installed ruby-sass and postgreSQL. I installed psycopg2 with both pip and pip3. I also installed gunicorn as the WSGI server for nginx. The nginx config is [1]. I configured gunicorn as an upstart job to run as user:group mailman:mailman [2] and set its config as recommended [3]. | I installed these on the third server. |
| Line 17: | Line 28: |
| The main departure from a bundler production install is that I wanted to install the various Mailman components (and dependencies) system wide from the heads of the respective gitlab branches. | Also, lm3o and mpo are both Ubuntu 14.04 and the native Python 3 is Python 3.4, so I had previously installed Python 3.6 from source. It turned out that I also couldn't install libapache2-mod-wsgi-py3 on mpo via apt because that installed a Python 3.4 version. Ultimately on mpo I installed mod_wsgi via pip, but this required re-running `configure` and `make install` for Python 3.6 because I hadn't specified `--enable-shared` initially and that is required to build mod_wsgi. |
| Line 19: | Line 30: |
| To this end, I cloned the hyperkitty, mailmanclient, postorius, mailman and mailman-hyperkitty branches into /opt/git. and ran their respective 'setup.py install' scripts with the appropriate system Python (3.4 for mailman and mailman-hyperkitty and 2.7 for the rest). The next step is the hardest. for those packages in mailman-bundler/eggs that had corresponding system package (probably in /usr/lib/python2.7/dist-packages) that could be imported in the system Python, I just deleted the mailman-bundler/eggs/ package and similarly for those Python 3 packages in mailman-bundler/venv-3.4/lib/python3.4/site-packeges that had a corresponding system package. | After some months, on mpo, Apache was upgraded by an automatic upgrade. This caused failures in mod_wsgi, probably because the upgraded Apache's mod_ssl used a different version of the SSL library than the one mod_wsgi was built with. See https://github.com/GrahamDumpleton/mod_wsgi/issues/370. However, before this was diagnosed, I switched mpo to use Gunicorn with Apache. |
| Line 21: | Line 32: |
| I also went through the mailman-bundler/bin/* scripts and removed now non-existent paths from the list added to sys.path. | The third server is Ubuntu 16.04 with Python 3.5, so it was OK as is. |
| Line 23: | Line 34: |
| There were some issues because I used only Django 1.9. django-haystack 2.4 is not compatible with Django 1.9, so I cloned the 2.5.dev branch from github and installed that. | === Installation === For these installs, I opted to install the entire Mailman suite in a virtualenv. The biggest motivation for this choice was the fact that there are already production lists on both servers and using a virtualenv allowed me to do a lot of the work in advance without interfering with the production install. Possibly influenced by these experiences, I am now using virtualenv for all installs. |
| Line 25: | Line 37: |
| There are also a lot of deprecation warnings in logs, some of which I've tried to address. | Note that all of this is done as the `mailman` user. |
| Line 27: | Line 39: |
| The remainder consisted of putting desired configuration stuff in mailman-bundler/mailman_web/settings_local.py [4], and editing mailman-bundler/deployment/mailman.cfg [5], mailman-bundler/deployment/mailman-hyperkitty.cfg [6] and mailman-bundler/mailman_web/urls.py [7]. | I already had some things set up in `/opt/mailman` including a `git` subdirectory containing clones of the !GitLab `mailman`, `mailmanclient`, `mailman-hyperkitty`, `hyperkitty`, `django-mailman3` and `postorius` projects. On the third server I have added `mailman-suite` because I use the settings.py from that project as the basis for mine. |
| Line 29: | Line 41: |
| I also tried to create an upstart script for mailman, but despite trying all options, I couldn't get it to get the proper demonized PID so I went with an init.d script [8]. | I then created a `/opt/mailman/mm` directory and within that a Python 3.6 virtualenv named `/opt/mailman/mm/venv`. I then activated the virtualenv and did |
| Line 31: | Line 43: |
| The Mailman specific Postfix things are very straightforward [9]. | `pip install psycopg2-binary`<<BR>> `pip install pylibmc`<<BR>> `pip install Whoosh` |
| Line 33: | Line 47: |
| For authentication, I couldn't get Google OpenID to work, but I was able to use OAuth2. I tried Twitter, but you can't get email address from Twitter so thats out. See [4] for details. Also, I added a bit to mailman-bundler/bin/mailman-post-update to do | on both servers. These are dependencies for using the PostgreSQL database, memcached and the Whoosh backend for !HyperKitty archive search. |
| Line 35: | Line 49: |
| cd $STATIC_DIR/hyperkitty/img/login ln -s google.png google-oauth2.png cd $OLDPWD |
I also did the same, but in a Python 3.5 virtualenv on the third server. |
| Line 39: | Line 51: |
| for the login graphic. | Then because lm3o uses nginx and Gunicorn to support wsgi apps, I did |
| Line 41: | Line 53: |
| Because there are | `pip install gunicorn` |
| Line 43: | Line 55: |
| there, and on mpo which uses Apache and mod_wsgi, I did | |
| Line 44: | Line 57: |
| The installation on mail.python.org was similar in many ways but different in others. The major differences are because mpo is an established server running Apache and Postfix and supporting over 300 Mailman 2.1 lists in the python.org domain which is a Postfix virtual alias domain. This complicated the Postfix configuration because we wanted Mailman 3 lists to also be in the python.org email domain and Postfix virtual alias domains don't honor transport maps which is the normal way of delivering mail from Postfix to MM 3. | `pip install mod_wsgi` |
| Line 46: | Line 59: |
| Another difference is I decided to use virtualenvs for this install. | but as noted above, mpo is now using Apache and Gunicorn like the third server. |
| Line 48: | Line 61: |
| Much of the install was the same. I have some notes of what was done, and I will only elaborate differences. | The third server uses Apache and Gunicorn so I did |
| Line 50: | Line 63: |
| Steps were: | `pip install gunicorn` |
| Line 52: | Line 65: |
| Create /opt/mailman owned by mailman. | there. |
| Line 54: | Line 67: |
| As the mailmanuser, git clone mailman-bundler. | === Migration === The prior install on both servers used Mailman Bundler which is obsolete, however for consistency I kept some of the same names. Everything is in the /opt/mailman/ directory. I maintained the prior git/ subdirectory along with two bash scripts, [[attachment:pull_script.txt|pull|view]] and [[attachment:build_script.txt|build|view]] which are used to pull changes from !GitLab and build everything in the venv. |
| Line 56: | Line 70: |
| Basically follow bundler docs. | I maintained this on the third server as well. |
| Line 58: | Line 72: |
| Run buildout as production. | === Recommended Configuration === |
| Line 60: | Line 74: |
| Install sass. | Here are my recommendations for the rest of the configuration. The older installs don't adhere to this exactly, but this is what I'm doing going forward. |
| Line 62: | Line 76: |
| Install apache mod_wsgi. The Apache config is [10] | Create directories: * /opt/mailman/mm/bin/ * /opt/mailman/mm/var/ and files: * /opt/mailman/mm/__init__.py (empty) * /opt/mailman/mm/mailman.cfg ([[attachment:mailman.cfg|sample|view]]) * /opt/mailman/mm/mailman-hyperkitty.cfg ([[attachment:mailman-hyperkitty.cfg|sample|view]]) * /opt/mailman/mm/settings_local.py ([[attachment:settings_local.py|sample|view]]) * /opt/mailman/mm/settings.py (copy of the mailman-suite settings.py file but with `DEBUG = False`) * /opt/mailman/mm/urls.py ([[attachment:urls.py|sample|view]]) * /opt/mailman/mm/wsgi.py ([[attachment:wsgi.py|sample|view]]) if using Gunicorn: * /opt/mailman/mm/gunicorn.conf ([[attachment:gunicorn.conf|sample|view]]) and a symlink: * /opt/mailman/mm/logs -> /opt/mailman/mm/var/logs In the /opt/mailman/mm/bin directory, create these executables: * /opt/mailman/mm/bin/django-admin ([[attachment:django-admin.txt|sample|view]]) Script to run Django management commands. * /opt/mailman/mm/bin/mailman ([[attachment:mailman.txt|sample|view]]) Script to run mailman commands. * /opt/mailman/mm/bin/mailman-post-update ([[attachment:mailman-post-update.txt|sample|view]]) Script to update static web and run migrations following a software update. if using Gunicorn: * /opt/mailman/mm/bin/gunicorn ([[attachment:gunicorn.txt|sample|view]]) Script to start Gunicorn. === MTA === I use Postfix and configuration is straightforward. Mailman's default configuration is appropriate for Postfix listening on 127.0.0.1:25. The main.cf settings are well documented at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#postfix. All that is really needed with recent postfix is |
| Line 64: | Line 100: |
| Make mailman_web/local_settings.py [4] with differences [11] | `recipient_delimiter = +` |
| Line 66: | Line 102: |
| Edit mailman_web/urls.py (change urlpatterns to list for django 1.9 and remove {"SSL": True\} [7] | and the additions to transport_maps, local_recipient_maps and relay_domains as described at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#transport-maps or, if you are using alias domains, the additions to transport_maps, virtual_alias_maps and relay_domains as described at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#unusual-postfix-configuration. === Web Server === As noted above the three servers I'm describing all have different web server configurations as follows: ===== nginx and Gunicorn ===== Here is the relevant nginx [[attachment:lm3o_nginx.txt|configuration|view]]. <<BR>> And one that redirects [[attachment:lm3o_nginx2.txt|http to https|view]]. ===== Apache and mod_wsgi ===== Here's an Apache [[attachment:mpo_wsgi.txt|configuration|view]] for mod_wsgi which I no longer use. ===== Apache and Gunicorn ===== Here's an Apache [[attachment:apache_gunicorn.txt|configuration|view]] for Gunicorn. === Running Things === Finally, we need to arrange that all the services are running. Generally, the OS has taken care of init, upstart or systemd scripts for Postfix, the web server, PostgreSQL and memcached, but other things that need to be run are Mailman core, Django's qcluster and Gunicorn if that is used. Also, there are periodic crons that need to be run. Here are some samples. |
| Line 68: | Line 116: |
| Edit base_url and api_key in /opt/mailman/mailman-bundler/deployment/mailman-hyperkitty.cfg [6] (values are different) | * Systemd [[attachment:systemd_mailman.txt|mailman.service|view]]. * Systemd [[attachment:systemd_qcluster.txt|qcluster.service|view]]. * Systemd [[attachment:systemd_gunicorn.txt|gunicorn.service|view]]. * Init [[attachment:init.d_mailman.txt|mailman init.d|view]]. * Upstart [[attachment:upstart_qcluster.txt|qcluster.conf|view]]. * Upstart [[attachment:upstart_gunicorn.txt|gunicorn.conf|view]]. * Cron [[attachment:mailman_crontab.txt|Mailman's crontab|view]]. |
| Line 70: | Line 124: |
| Install postgresql create mailmanweb DB and mailman user. Install postgresql-server-dev-9.3 and then pip install psycopg2 in both venvs. Make /opt/mailman/git/ and clone django-haystack mailman mailman-hyperkitty hyperkitty mailmanclient postorius repos and run the setups in the appropriate venv. Remove older mailman and mailman-hyperkitty from venv-3.4/lib/... In eggs/ for everything with a version in venv/lib/..., remove the eggs/ subdir and replace it with a symlink to the venv/lib/... version. Link both Django 1.8 and 1.9 to the Django 1.9 in venv/lib/... Create django superuser. Update Oauth providers to recognize the additional callback domain. A major issue on mpo was trying to have MM 3 lists have @python.org email addresses so these wouldn't need to change upon list migration. In cases where the list domain is in Postfix mydestinations, this is straightforward, but when the list domain is a virtual alias domain, it's more difficult. What I did is a bit of a kludge. We plan to make this a feature so it too will be straightforward, but it isn't there yet. The solution is as follows: Use virtual_alias_maps to map list(-*)@python.org to list(-*)@x.python.org, and then use transport_maps to relay list(-*)@x.python.org to Mailman's LMTP runner in the usual way, and finally modify Mailman's LMTP runner to rewrite the x.python.org recipient domain to python.org so the address corresponds to what Mailman expects. The details involve several pieces. Add a section [12] to mailman-bundler/deployment/mailman.cfg to reference mailman-bundler/deployment/postfix.cfg [13] which in turn sets postmap_command to mailman-bundler/bin/postmap_command [14] which creates additional files. Reference these files in Postfix [15] and patch mailman/runners/lmtp.py [16]. [1] This is the nginx config for Mailman. The configs for http and https are essentially the same except for the added SSL stuff for https. The critical stuff is the /static/ location and the proxy stuff for proxying to gunicorn at port 8000. server { listen 80 default_server; # listen [::]:80 default_server ipv6only=on; root /usr/share/nginx/html; index index.html index.htm; server_name lists.mailman3.org; location / { proxy_pass http://127.0.0.1:8000/; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; } location /static/ { alias /var/spool/mailman-web/static/; } location /robots.txt { alias /var/spool/mailman-web/static/hyperkitty/robots.txt; } location /favicon.ico { alias /var/www/listorg/images/favicon.ico; } } server { listen 443 default_server; # listen [::]:443 default_server ipv6only=on; root /usr/share/nginx/html; index index.html index.htm; server_name lists.mailman3.org; ssl on; ssl_certificate /etc/letsencrypt/live/mirror.mailman3.org/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/mirror.mailman3.org/privkey.pem; ssl_session_timeout 5m; ssl_protocols SSLv3 TLSv1 TLSv1.1 TLSv1.2; ssl_ciphers "HIGH:!aNULL:!MD5 or HIGH:!aNULL:!MD5:!3DES"; ssl_prefer_server_ciphers on; location / { proxy_pass http://127.0.0.1:8000/; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Host $host; proxy_set_header X-Forwarded-Server $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_redirect off; } location /static/ { alias /var/spool/mailman-web/static/; } location /robots.txt { alias /var/spool/mailman-web/static/hyperkitty/robots.txt; } location /favicon.ico { alias /var/www/listorg/images/favicon.ico; } } [2] gunicorn upstart (/etc/init/gunicorn.conf) description "gunicorn for Mailman" start on runlevel [2345] stop on runlevel [016] respawn respawn limit 10 5 setuid mailman setgid mailman chdir /opt/mailman/mailman-bundler exec /opt/mailman/mailman-bundler/bin/gunicorn -c /opt/mailman/mailman-bundler/deployment/gunicorn.conf mailman_web.wsgi:application {3] gunicorn config (/opt/mailman/mailman-bundler/deployment/gunicorn.conf) bind = ['127.0.0.1:8000'] proc_name = "mailman-web" chdir = "/opt/mailman/mailman-bundler" pidfile = "/opt/mailman/mailman-bundler/var/gunicorn.pid" accesslog = "/opt/mailman/mailman-bundler/var/logs/access.log" errorlog = "/opt/mailman/mailman-bundler/var/logs/error.log" #daemon = True [4] mailman-bundler/mailman_web/settings_local.py (not the real keys/secrets) # SECURITY WARNING: keep the secret key used in production secret! SECRET_KEY = 'django secret' MAILMAN_ARCHIVER_KEY = 'arch_secret' ADMINS = ( ('Mailman Admin', 'mailman@localhost'), ) # Hosts/domain names that are valid for this site; required if DEBUG is False # See https://docs.djangoproject.com/en/1.9/ref/settings/#allowed-hosts #ALLOWED_HOSTS = ['*'] ALLOWED_HOSTS = ['localhost', '127.0.0.1', 'lists.mailman3.org', 'mail.mailman3.org', 'mailman.iad1.psf.io', 'mail.falconframework.org', '104.239.228.201', ] # And for BrowserID too, see # http://django-browserid.rtfd.org/page/user/settings.html#django.conf.settings.BROWSERID_AUDIENCES BROWSERID_AUDIENCES = [ "http://localhost", "http://localhost:8000", "http://127.0.0.1:8000", "http://lists.mailman3.org", "http://mail.falconframework.org", "https://localhost", "https://localhost:8000", "https://127.0.0.1:8000", "https://lists.mailman3.org", "https://mail.falconframework.org", ] ## Email confirmation / address activation # Add a from-address for email-confirmation: # EMAIL_CONFIRMATION_FROM = 'postmaster@example.org' EMAIL_CONFIRMATION_FROM = 'mailman@mailman3.org' MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1', '::ffff:127.0.0.1', '104.239.228.201', ) DATABASES = { 'default': { 'ENGINE': 'django.db.backends.postgresql_psycopg2', # Last part is one of 'postgresql_psycopg2', 'mysql', 'sqlite3' or 'oracle'. 'NAME': 'mailmanweb', # Example, change as needed 'USER': 'mailman', 'PASSWORD': 'db_passwd', 'HOST': '127.0.0.1', # Empty for localhost through domain sockets or '127.0.0.1' for localhost through TCP. 'PORT': '5432', # Set to empty string for default. } } # If you're behind a proxy, use the X-Forwarded-Host header # See https://docs.djangoproject.com/en/1.8/ref/settings/#use-x-forwarded-host USE_X_FORWARDED_HOST = True # And if your proxy does your SSL encoding for you, set SECURE_PROXY_SSL_HEADER # see https://docs.djangoproject.com/en/1.5/ref/settings/#secure-proxy-ssl-header SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https') # Use SSL when logged in. You need to enable the SSLRedirect middleware for # this feature to work. USE_SSL = True TIME_ZONE = 'UTC' AUTHENTICATION_BACKENDS = ( #'social.backends.open_id.OpenIdAuth', # http://python-social-auth.readthedocs.org/en/latest/backends/google.html #'social.backends.google.GoogleOpenId', 'social.backends.google.GoogleOAuth2', # Twitter isn't working ... # Rumor has it you can't get email from twitter :( #'social.backends.twitter.TwitterOAuth', # Yahoo works, but don't encourage Yahoo until we get DMARC mitigations. #'social.backends.yahoo.YahooOpenId', 'social.backends.facebook.FacebookOAuth2', 'django_browserid.auth.BrowserIDBackend', 'django.contrib.auth.backends.ModelBackend', ) SOCIAL_AUTH_TWITTER_KEY = 'twitter key' SOCIAL_AUTH_TWITTER_SECRET = 'twitter secret' SOCIAL_AUTH_TWITTER_SCOPE = ['email', 'name'] SOCIAL_AUTH_GOOGLE_OAUTH2_KEY = 'google key' SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET = 'google secret' SOCIAL_AUTH_GOOGLE_OAUTH2_USE_DEPRECATED_API = True SOCIAL_AUTH_FACEBOOK_KEY = 'fb key' SOCIAL_AUTH_FACEBOOK_SECRET = 'fb secret' SOCIAL_AUTH_FACEBOOK_SCOPE = ['email', 'public_profile'] SOCIAL_AUTH_FACEBOOK_PROFILE_EXTRA_PARAMS = { 'fields': 'name, email', } [5] mailman-bundler/deployment/mailman.cfg changes are site_owner, [database] and [shell] # This is the absolute bare minimum base configuration file. User supplied # configurations are pushed onto this. [mailman] # This address is the "site owner" address. Certain messages which must be # delivered to a human, but which can't be delivered to a list owner (e.g. a # bounce from a list owner), will be sent to this address. It should point to # a human. site_owner: mailman@mailman3.org layout: here [paths.here] # Everything in the same directory var_dir: /opt/mailman/mailman-bundler/var [database] class: mailman.database.postgresql.PostgreSQLDatabase #url: postgres://mailman:mailman-db-password@localhost/mailman url: postgres://mailman:db_passwd@localhost/mailmanweb [archiver.hyperkitty] class: mailman_hyperkitty.Archiver enable: yes configuration: /opt/mailman/mailman-bundler/deployment/mailman-hyperkitty.cfg [archiver.prototype] enable: yes [shell] history_file: $var_dir/history.py #[logging.database] #level: debug [6] mailman-bundler/deployment/mailman-hyperkitty.cfg changes are base_url and api_key # This is the mailman extension configuration file to enable HyperKitty as an # archiver. Remember to add the following lines in the mailman.cfg file: # # [archiver.hyperkitty] # class: mailman_hyperkitty.Archiver # enable: yes # configuration: /path/to/here/hyperkitty.cfg # [general] base_url: https://lists.mailman3.org/archives api_key: arch_secret [7] mailman-bundler/mailman_web/urls.py Changes are converting url patterns to a list for Django 1.9 and removing '{"SSL": True\} from a couple of URLs as it caused problems. from django.conf.urls import include, url from django.core.urlresolvers import reverse_lazy from django.views.generic import RedirectView # Comment the next two lines to disable the admin: from django.contrib import admin admin.autodiscover() urlpatterns = [url(r'^$', RedirectView.as_view(url=reverse_lazy('hyperkitty.views.index.index'))), url(r'^mailman3/', include('postorius.urls')), url(r'^archives/', include('hyperkitty.urls')), url(r'', include('social.apps.django_app.urls', namespace='social')), url(r'', include('django_browserid.urls')), ] [8] /etc/init.d/mailman ### BEGIN INIT INFO # Provides: GNU Mailman # Required-Start: $local_fs $network # Required-Stop: $local_fs $network # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Mailman Service # Description: service control for Mailman ### END INIT INFO PATH=/opt/mailman/mailman-bundler/bin:/usr/sbin:/usr/bin:/bin:/sbin: DESC="GNU Mailman service" DAEMON=/opt/mailman/mailman-bundler/bin/mailman NAME=mailman USER=mailman GROUP=mailman # Exit if the package is not installed [ -x "$DAEMON" ] || exit 0 # Load the VERBOSE setting and other rcS variables . /lib/init/vars.sh # Define LSB log_* functions. # Depend on lsb-base (>= 3.2-14) to ensure that this file is present # and status_of_proc is working. . /lib/lsb/init-functions case "$1" in start) [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME" sudo -u $USER $DAEMON start ;; stop) [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME" sudo -u $USER $DAEMON stop ;; status) sudo -u $USER $DAEMON status ;; reopen) sudo -u $USER $DAEMON reopen ;; restart) log_daemon_msg "Restarting $DESC" "$NAME" sudo -u $USER $DAEMON restart ;; *) echo "Usage: $SCRIPTNAME {start|stop|status|reopen|restart}" >&2 exit 3 ;; esac [9] MM 3 specific Postfix additions lm3o local_recipient_maps = hash:/opt/mailman/mailman-bundler/var/data/postfix_lmtp proxy:unix:passwd.byname $alias_maps relay_domains = hash:/opt/mailman/mailman-bundler/var/data/postfix_domains transport_maps = hash:/opt/mailman/mailman-bundler/var/data/postfix_lmtp [10] Apache mod_wsgi config. This goes in a global section. WSGIDaemonProcess mailman-web display-name=mailman-web maximum-requests=1000 umask=0002 user=mailman group=mailman python-path=/opt/mailman/mailman-bundler/venv/lib/python2.7/site-packages:/opt/mailman/mailman-bundler/eggs home=/opt/mailman/mailman-bundler/var WSGIRestrictSignal Off (I'm not sure that WSGIRestrictSignal Off is required, but it was in the provided example so I kept it. I also made changes to WSGIDaemonProcess based on my own mod_wsgi experience elsewhere.) This goes in the VirtualHost block for the domain. # Mailman 3 stuff Alias /static "/var/spool/mailman-web/static" <Directory "/var/spool/mailman-web/static"> Require all granted </Directory> WSGIScriptAlias /mm3 /opt/mailman/mailman-bundler/bin/mailman-web.wsgi <Directory "/opt/mailman/mailman-bundler/bin"> <Files mailman-web.wsgi> Order deny,allow Allow from all Require all granted </Files> WSGIProcessGroup mailman-web </Directory> [11] Differences between lm3o settings_local.py [4] and mpo. ALLOWED_HOSTS = ['localhost', '127.0.0.1', 'mail.python.org' ] BROWSERID_AUDIENCES = [ "http://localhost", "http://mail.python.org", "https://localhost", "https://mail.python.org", ] EMAIL_CONFIRMATION_FROM = 'mailman@python.org' MAILMAN_ARCHIVER_FROM = ('127.0.0.1', '::1', '::ffff:127.0.0.1', '188.166.95.178', ) [12] Addition to mailman.cfg for Postfix kludge [mta] configuration: /opt/mailman/mailman-bundler/deployment/postfix.cfg [13] Content of postfix.cfg [postfix] # Additional configuration variables for the postfix MTA. # This variable describe the program to use for regenerating the transport map # db file, from the associated plain text files. The file being updated will # be appended to this string (with a separating space), so it must be # appropriate for os.system(). postmap_command: /opt/mailman/mailman-bundler/bin/postmap_command [14] Content of postmap_command script. #!/bin/bash umask 0002 LMTP=/opt/mailman/mailman-bundler/var/data/postfix_lmtp XLMTP=/opt/mailman/mailman-bundler/var/data/postfix_xlmtp VMAP=/opt/mailman/mailman-bundler/var/data/postfix_vmap DOM=/opt/mailman/mailman-bundler/var/data/postfix_domains XDOM=/opt/mailman/mailman-bundler/var/data/postfix_xdomains if [[ $1 =~ postfix_lmtp$ ]] ; then sed < $LMTP > $XLMTP -e 's/@python.org/@x.python.org/' sed < $LMTP > $VMAP -r -e 's/(^[^ ]*)@python.org( *).*/\1@python.org\2\1@x.python.org/' /usr/sbin/postmap $XLMTP /usr/sbin/postmap $VMAP elif [[ $1 =~ postfix_domains$ ]] ; then sed < $DOM > $XDOM -e 's/^python.org/x.python.org/' /usr/sbin/postmap $XDOM else /usr/sbin/postmap $1 fi [15] Postfix main.cf additions for mpo. relay_domains = hash:/opt/mailman/mailman-bundler/var/data/postfix_xdomains transport_maps = cdb:/etc/postfix/transport hash:/opt/mailman/mailman-bundler/var/data/postfix_xlmtp virtual_alias_maps = cdb:/etc/postfix/virtual-python cdb:/etc/postfix/virtual-snakebite cdb:/etc/postfix/virtual-usergroups cdb:/etc/postfix/virtual-raspberry hash:/srv/mailman/data/virtual-mailman hash:/opt/mailman/mailman-bundler/var/data/postfix_vmap Note that the order of hash:/srv/mailman/data/virtual-mailman (MM2 maps) and hash:/opt/mailman/mailman-bundler/var/data/postfix_vmap (MM3 maps) determines in the case of MM2 and MM3 lists with the same name, which list gets the mail. [16] lmtp/py patch --- git/mailman/src/mailman/runners/lmtp.py 2016-06-05 16:58:06.255217017 -0400 +++ mailman-bundler/venv-3.4/lib/python3.4/site-packages/mailman-3.1.0-py3.4.egg/mailman/runners/lmtp.py 2016-06-21 00:47:29.497817768 -0400 @@ -118,6 +118,9 @@ subaddress may be None if this is the list's posting address. """ localpart, domain = address.split('@', 1) + # Kludge for x.python.org <-> python.org + if domain == 'x.python.org': + domain = 'python.org' localpart = localpart.split(config.mta.verp_delimiter, 1)[0] parts = localpart.split(DASH) if parts[-1] in SUBADDRESS_NAMES: |
<<IncludeComments>> |
Introduction
I have now installed Mailman 3 for production on two different servers. This is intended to be documentation of my decisions and experiences in doing so. It will be more of a narrative than a how-to, but I hope a how-to can be distilled from it.
The two servers are lists.mailman3.org (lm3o) and mail.python.org (mpo). Pre-existing things on those servers forced some decisions which were different between them, but the actual MM 3 installation was pretty much up to me, yet I did it differently the second time due to things I thought I'd learned the first time.
A fundamental step in any installation is to ensure that you have the necessary infrastructure for reliably sending mail. This mostly involves DNS records. I won't go into detail, but you need an A and maybe AAAA record for your MAIL FROM domain and rDNS PTR records for the IPs pointing back to the domain. You may also need an MX record for your email domain pointing to the Mailman server. Also good is a mechanism for DKIM signing outgoing mail and DKIM and SPF records for the domain.
This already existed on mpo, but needed to be set up on lm3o.
lm3o is the same server as mirror.list.org/mirror.mailman3.org. This is a mirror of the GNU Mailman web site and was already set up using nginx as a web server. Thus, the MM 3 installation here uses nginx as the web server proxying to gunicorn for WSGI support.
Update
I have done a significant reinstallation on both servers. The prior version of this page is no longer applicable to these servers. I have also installed MM 3 on a third server. I have now evolved my process and installations to a stable point. It is this configuration that I will describe.
The information about templates in the lists.mailman3.org section in the prior version of this page is still valid, but the path /opt/mailman/mailman-bundler/var/templates/... becomes /opt/mailman/mm/var/templates/...
Preliminaries
Both lm3o and mpo already had sass and memcached installed via
apt install ruby-sass
apt install memcached
I installed these on the third server.
Also, lm3o and mpo are both Ubuntu 14.04 and the native Python 3 is Python 3.4, so I had previously installed Python 3.6 from source. It turned out that I also couldn't install libapache2-mod-wsgi-py3 on mpo via apt because that installed a Python 3.4 version. Ultimately on mpo I installed mod_wsgi via pip, but this required re-running configure and make install for Python 3.6 because I hadn't specified --enable-shared initially and that is required to build mod_wsgi.
After some months, on mpo, Apache was upgraded by an automatic upgrade. This caused failures in mod_wsgi, probably because the upgraded Apache's mod_ssl used a different version of the SSL library than the one mod_wsgi was built with. See https://github.com/GrahamDumpleton/mod_wsgi/issues/370. However, before this was diagnosed, I switched mpo to use Gunicorn with Apache.
The third server is Ubuntu 16.04 with Python 3.5, so it was OK as is.
Installation
For these installs, I opted to install the entire Mailman suite in a virtualenv. The biggest motivation for this choice was the fact that there are already production lists on both servers and using a virtualenv allowed me to do a lot of the work in advance without interfering with the production install. Possibly influenced by these experiences, I am now using virtualenv for all installs.
Note that all of this is done as the mailman user.
I already had some things set up in /opt/mailman including a git subdirectory containing clones of the GitLab mailman, mailmanclient, mailman-hyperkitty, hyperkitty, django-mailman3 and postorius projects. On the third server I have added mailman-suite because I use the settings.py from that project as the basis for mine.
I then created a /opt/mailman/mm directory and within that a Python 3.6 virtualenv named /opt/mailman/mm/venv. I then activated the virtualenv and did
pip install psycopg2-binary
pip install pylibmc
pip install Whoosh
on both servers. These are dependencies for using the PostgreSQL database, memcached and the Whoosh backend for HyperKitty archive search.
I also did the same, but in a Python 3.5 virtualenv on the third server.
Then because lm3o uses nginx and Gunicorn to support wsgi apps, I did
pip install gunicorn
there, and on mpo which uses Apache and mod_wsgi, I did
pip install mod_wsgi
but as noted above, mpo is now using Apache and Gunicorn like the third server.
The third server uses Apache and Gunicorn so I did
pip install gunicorn
there.
Migration
The prior install on both servers used Mailman Bundler which is obsolete, however for consistency I kept some of the same names. Everything is in the /opt/mailman/ directory. I maintained the prior git/ subdirectory along with two bash scripts, pull and build which are used to pull changes from GitLab and build everything in the venv.
I maintained this on the third server as well.
Recommended Configuration
Here are my recommendations for the rest of the configuration. The older installs don't adhere to this exactly, but this is what I'm doing going forward.
Create directories:
- /opt/mailman/mm/bin/
- /opt/mailman/mm/var/
and files:
/opt/mailman/mm/init.py (empty)
/opt/mailman/mm/mailman.cfg (sample)
/opt/mailman/mm/mailman-hyperkitty.cfg (sample)
/opt/mailman/mm/settings_local.py (sample)
/opt/mailman/mm/settings.py (copy of the mailman-suite settings.py file but with DEBUG = False)
/opt/mailman/mm/urls.py (sample)
/opt/mailman/mm/wsgi.py (sample)
if using Gunicorn:
/opt/mailman/mm/gunicorn.conf (sample)
and a symlink:
/opt/mailman/mm/logs -> /opt/mailman/mm/var/logs
In the /opt/mailman/mm/bin directory, create these executables:
/opt/mailman/mm/bin/django-admin (sample) Script to run Django management commands.
/opt/mailman/mm/bin/mailman (sample) Script to run mailman commands.
/opt/mailman/mm/bin/mailman-post-update (sample) Script to update static web and run migrations following a software update.
if using Gunicorn:
/opt/mailman/mm/bin/gunicorn (sample) Script to start Gunicorn.
MTA
I use Postfix and configuration is straightforward. Mailman's default configuration is appropriate for Postfix listening on 127.0.0.1:25. The main.cf settings are well documented at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#postfix. All that is really needed with recent postfix is
recipient_delimiter = +
and the additions to transport_maps, local_recipient_maps and relay_domains as described at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#transport-maps or, if you are using alias domains, the additions to transport_maps, virtual_alias_maps and relay_domains as described at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#unusual-postfix-configuration.
Web Server
As noted above the three servers I'm describing all have different web server configurations as follows:
nginx and Gunicorn
Here is the relevant nginx configuration.
And one that redirects http to https.
Apache and mod_wsgi
Here's an Apache configuration for mod_wsgi which I no longer use.
Apache and Gunicorn
Here's an Apache configuration for Gunicorn.
Running Things
Finally, we need to arrange that all the services are running. Generally, the OS has taken care of init, upstart or systemd scripts for Postfix, the web server, PostgreSQL and memcached, but other things that need to be run are Mailman core, Django's qcluster and Gunicorn if that is used. Also, there are periodic crons that need to be run. Here are some samples.
Systemd mailman.service.
Systemd qcluster.service.
Systemd gunicorn.service.
Init mailman init.d.
Upstart qcluster.conf.
Upstart gunicorn.conf.
Cron Mailman's crontab.
Comments
This was written assuming that the reader is familiar with Python software installation and tools like git and virtualenv.
Also, please note the first paragraph of the introduction. This is documentation of my decisions and experiences. It is not a how-to and does not replace the documentation at http://docs.list.org/
I've been asked to provide more information about some of the things that may be unfamiliar to some users.
In the Migration section, I refer to a /opt/mailman/git/ directory containing clones of the various GitLab projects and the 'pull' and 'build' scripts to pull and install updates. First, you need git which can be installed via your OS package manager if you don't have it. The cloned projects themselves are created in the /opt/mailman/git/ directory. This is done using one of two git commands for each project. If you have a gitlab account with your SSH key(s), you can
cd /opt/mailman/git/
git clone git@gitlab.com:mailman/project_name.git
for each project. Otherwise, you can
cd /opt/mailman/git/
git clone https://gitlab.com/mailman/project_name.git
I also mention in the Installation section, creating a Python 3.6 virtualenv. Python 3.6 is currently the minimum required. It could just as well be any later Python 3 version and should be the latest available on your server.
Creating the /opt/mailman/mm/venv virtualenv is done by
cd /opt/mailman/mm
virtualenv venv
or if you want to specify a Python other than the default
cd /opt/mailman/mm
virtualenv -p /path/to/desired/python venv
This is then activated by
source /opt/mailman/mm/venv/bin/activate
When active in a particular shell, it can be deactivated with the command
deactivate
or by exiting that shell.
If you don't have virtualenv, it can be installed with pip or via your OS package manager. For the latter, it might be a package named virtualenv or python3-virtualenv.