Differences between revisions 2 and 11 (spanning 9 versions)
Revision 2 as of 2016-07-03 23:36:53
Size: 8633
Editor: msapiro
Comment: Starting to document lm3o and mpo MM3 installs.
Revision 11 as of 2018-05-27 21:42:39
Size: 1690
Editor: msapiro
Comment: Prepare for major update.
Deletions are marked like this. Additions are marked like this.
Line 12: Line 12:
For both installs I started with mailman-bundler.
=== lists.mailman3.org ===
On lm3o, I created /opt/mailman and cloned mailman-bundler into /opt/mailman/mailman-bundler. I more or less followed its docs for a production install, but I decided not to use virtualenvs, so I set buildout.cfg for 'production' and ran buildout not in a virtualenv - also I probably had to ensure I had up to date setuptools and pip.

I installed ruby-sass and postgreSQL. I installed psycopg2 with both pip and pip3. I also installed gunicorn as the WSGI server for nginx and [[attachment:lm3o_nginx.txt|configured]] ngnix. I configured gunicorn as an [[attachment:lm3o_gunicorn.txt|upstart job]] to run as user:group mailman:mailman and set its [[attachment:lm3o_gunicorn.conf.txt|config]] as recommended.

The main departure from a bundler production install is that I wanted to install the various Mailman components (and dependencies) system wide from the heads of the respective gitlab branches.

To this end, I cloned the hyperkitty, mailmanclient, postorius, mailman and mailman-hyperkitty branches into /opt/git. and ran their respective 'setup.py install' scripts with the appropriate system Python (3.4 for mailman and mailman-hyperkitty and 2.7 for the rest). The next step is the hardest. For those packages in mailman-bundler/eggs that had corresponding system package (probably in /usr/lib/python2.7/dist-packages) that could be imported in the system Python, I just deleted the mailman-bundler/eggs/ package and similarly for those Python 3 packages in mailman-bundler/venv-3.4/lib/python3.4/site-packeges that had a corresponding system package.

I also went through the mailman-bundler/bin/* scripts and removed now non-existent paths from the list added to sys.path.

There were some issues because I used only Django 1.9. django-haystack 2.4 is not compatible with Django 1.9, so I cloned the django-haystack 2.5.dev branch from github and installed that.

There are also a lot of deprecation warnings in logs, some of which I've tried to address.

The remainder consisted of putting desired configuration in [[attachment:lm3o_settings_local.txt|mailman-bundler/mailman_web/settings_local.py]], and editing [[attachment:lm3o_mailman.cfg.txt|mailman-bundler/deployment/mailman.cfg]], [[attachment:lm3o_hyperkitty.cfg.txt|mailman-bundler/deployment/mailman-hyperkitty.cfg]] and [[attachment:lm3o_urls.txt|mailman-bundler/mailman_web/urls.py]].

I also tried to create an upstart script for mailman, but despite trying all options, I couldn't get it to get the proper demonized PID so I went with an [[attachment:mailman_init.txt|init.d script]].

The Mailman specific Postfix [[attachment:lm3o_postfix.txt|things]] are very straightforward.

For authentication, I couldn't get Google OpenID to work, but I was able to use OAuth2. I tried Twitter, but you can't get email address from Twitter so that's out. See [[attachment:lm3o_settings_local.txt|settings_local.py]] for details. Also, I added a bit to mailman-bundler/bin/mailman-post-update to do
{{{
    cd $STATIC_DIR/hyperkitty/img/login
    ln -s google.png google-oauth2.png
    cd $OLDPWD
}}}
for the login graphic.

There are several default templates that have incorrect or generic URLs for the web UI. To make these better, I installed these templates in /opt/mailman/mailman-bundler/templates/site/en/:
 * [[attachment:footer-generic.txt]]
 * [[attachment:masthead.txt]]
 * [[attachment:newlist.txt]]
 * [[attachment:probe.txt]]
 * [[attachment:welcome.txt]]
and for each supported domain I installed in /opt/mailman/mailman-bundler/templates/domains/EMAIL_DOMAIN/en/:
 * [[attachment:postack.txt]]
with the literal `EMAIL_DOMAIN` replaced by the actual domain name.
=== mail.python.org ===
The installation on mail.python.org was similar in many ways but different in others. The major differences are because mpo is an established server running Apache and Postfix and supporting over 300 Mailman 2.1 lists in the python.org domain which is a Postfix virtual alias domain. This complicated the Postfix configuration because we wanted Mailman 3 lists to also be in the python.org email domain and Postfix virtual alias domains don't honor transport maps which is the normal way of delivering mail from Postfix to MM 3.

Another difference is I decided to use virtualenvs for this install.

Much of the install was the same. I have some notes of what was done, and I will only elaborate differences.

Steps were:

Create /opt/mailman owned by mailman.

As the mailmanuser, git clone mailman-bundler.

Basically follow bundler docs.

Run buildout as production.

Install ruby-sass.

Install apache mod_wsgi. The Apache wsgi config is [[attachment:mpo_wsgi.txt]].

Make [[attachment:lm3o_settings_local.txt|mailman_web/local_settings.py]] with [[attachment:mpo_settings_diff.txt|differences]]

Edit [[attachment:lm3o_urls.txt|mailman_web/urls.py]] (change urlpatterns to list for django 1.9 and remove `{"SSL": True\}`

Edit base_url and api_key in [[attachment:lm3o_hyperkitty.cfg.txt|/opt/mailman/mailman-bundler/deployment/mailman-hyperkitty.cfg]] (values are different).

Install postgresql create mailmanweb DB and mailman user.

Install postgresql-server-dev-9.3 and then pip install psycopg2 in both venvs.

Make /opt/mailman/git/ and clone django-haystack mailman mailman-hyperkitty hyperkitty mailmanclient postorius repos and run the setups in the appropriate venv.

Remove older mailman and mailman-hyperkitty from venv-3.4/lib/...

In eggs/ for everything with a version in venv/lib/..., remove the eggs/ subdir and replace it with a symlink to the venv/lib/... version. Link both Django 1.8 and 1.9 to the Django 1.9 in venv/lib/...

Create django superuser.

Update Oauth providers to recognize the additional callback domain.

Add custom templates as above for lm3o.

A major issue on mpo was trying to have MM 3 lists have @python.org email addresses so these wouldn't need to change upon list migration. In cases where the list domain is in Postfix mydestinations, this is straightforward, but when the list domain is a virtual alias domain, it's more difficult. What I did is a bit of a kludge. We plan to make this a feature so it too will be straightforward, but it isn't there yet.

The solution is as follows:

Use virtual_alias_maps to map `list(-*)@python.org` to `list(-*)@x.python.org`, and then use transport_maps to relay `list(-*)@x.python.org` to Mailman's LMTP runner in the usual way, and finally modify Mailman's LMTP runner to rewrite the `x.python.org` recipient domain to `python.org` so the address corresponds to what Mailman expects.

The details involve several pieces. Add a [[attachment:mpo_mailman.cfg.add.txt|section]] to mailman-bundler/deployment/mailman.cfg to reference [[attachment:mpo_postfix.cfg.txt|mailman-bundler/deployment/postfix.cfg]] which in turn sets postmap_command to [[attachment:mpo_postmap.txt|mailman-bundler/bin/postmap_command]] which creates additional files. Reference these files in [[attachment:mpo_postfix_main.txt|Postfix]] and [[attachment:mpo_lmtp_patch.txt|patch]] mailman/runners/lmtp.py.
=== Update ===
I have done a significant reinstallation on both servers. <<Action(recall,The prior version of this page,rev=10)>> is no longer applicable to these servers. I will be updating this page with more information about the current installs as time goes forward.

Introduction

I have now installed Mailman 3 for production on two different servers. This is intended to be documentation of my decisions and experiences in doing so. It will be more of a narrative than a how-to, but I hope a how-to can be distilled from it.

The two servers are lists.mailman3.org (lm3o) and mail.python.org (mpo). Pre-existing things on those servers forced some decisions which were different between them, but the actual MM 3 installation was pretty much up to me, yet I did it differently the second time due to things I thought I'd learned the first time.

A fundamental step in any installation is to ensure that you have the necessary infrastructure for reliably sending mail. This mostly involves DNS records. I won't go into detail, but you need an A and maybe AAAA record for your MAIL FROM domain and rDNS PTR records for the IPs pointing back to the domain. You may also need an MX record for your email domain pointing to the Mailman server. Also good is a mechanism for DKIM signing outgoing mail and DKIM and SPF records for the domain.

This already existed on mpo, but needed to be set up on lm3o.

lm3o is the same server as mirror.list.org/mirror.mailman3.org. This is a mirror of the GNU Mailman web site and was already set up using nginx as a web server. Thus, the MM 3 installation here uses nginx as the web server proxying to gunicorn for WSGI support.

Update

I have done a significant reinstallation on both servers. The prior version of this page is no longer applicable to these servers. I will be updating this page with more information about the current installs as time goes forward.

MailmanWiki: DOC/Mailman 3 installation experience (last edited 2023-11-24 16:16:40 by msapiro)