Added reference to our virtualenv install and credited Brian and Kelly
|Deletions are marked like this.||Additions are marked like this.|
|Line 2:||Line 2:|
|I have now installed Mailman 3 for production on two different servers. This is intended to be documentation of my decisions and experiences in doing so. It will be more of a narrative than a how-to, but I hope a how-to can be distilled from it. Also see our [[https://docs.mailman3.org/en/latest/install/virtualenv.html|virtualenv installation document]], Brian Carpenter's [[DOC/Howto_Install_Mailman3_On_Debian10|Howto_Install_Mailman3_On_Debian10]] and Kelly Close's [[https://docs.google.com/document/d/1xIcSsoNFp2nHi7r4eQys00s9a0k2sHhu1V5PlantPTs/edit|Mailman 3 Setup Using Ubuntu Bionic]]||I have now installed Mailman 3 for production on two different servers. This is intended to be documentation of my decisions and experiences in doing so. It will be more of a narrative than a how-to, but I hope a how-to can be distilled from it. Also see our [[https://docs.mailman3.org/en/latest/install/virtualenv.html|virtualenv installation document]], Brian Carpenter's [[DOC/Howto_Install_Mailman3_On_Debian10|Howto_Install_Mailman3_On_Debian10]] and Kelly Close's [[https://docs.google.com/document/d/1xIcSsoNFp2nHi7r4eQys00s9a0k2sHhu1V5PlantPTs/|Mailman 3 Setup Using Ubuntu Bionic]]|
I have now installed Mailman 3 for production on two different servers. This is intended to be documentation of my decisions and experiences in doing so. It will be more of a narrative than a how-to, but I hope a how-to can be distilled from it. Also see our virtualenv installation document, Brian Carpenter's Howto_Install_Mailman3_On_Debian10 and Kelly Close's Mailman 3 Setup Using Ubuntu Bionic
The two servers are lists.mailman3.org (lm3o) and mail.python.org (mpo). Pre-existing things on those servers forced some decisions which were different between them, but the actual MM 3 installation was pretty much up to me, yet I did it differently the second time due to things I thought I'd learned the first time.
A fundamental step in any installation is to ensure that you have the necessary infrastructure for reliably sending mail. This mostly involves DNS records. I won't go into detail, but you need an A and maybe AAAA record for your MAIL FROM domain and rDNS PTR records for the IPs pointing back to the domain. You may also need an MX record for your email domain pointing to the Mailman server. Also good is a mechanism for DKIM signing outgoing mail and DKIM and SPF records for the domain.
This already existed on mpo, but needed to be set up on lm3o.
lm3o is the same server as mirror.list.org/mirror.mailman3.org. This is a mirror of the GNU Mailman web site and was already set up using nginx as a web server. Thus, the MM 3 installation here uses nginx as the web server proxying to gunicorn for WSGI support.
I have done a significant reinstallation on both servers. The prior version of this page is no longer applicable to these servers. I have also installed MM 3 on a third server. I have now evolved my process and installations to a stable point. It is this configuration that I will describe.
Both lm3o and mpo already had sass and memcached installed via
apt install ruby-sass
apt install memcached
I installed these on the third server.
Also, lm3o and mpo are both Ubuntu 14.04 and the native Python 3 is Python 3.4, so I had previously installed Python 3.6 from source. It turned out that I also couldn't install libapache2-mod-wsgi-py3 on mpo via apt because that installed a Python 3.4 version. Ultimately on mpo I installed mod_wsgi via pip, but this required re-running configure and make install for Python 3.6 because I hadn't specified --enable-shared initially and that is required to build mod_wsgi.
After some months, on mpo, Apache was upgraded by an automatic upgrade. This caused failures in mod_wsgi, probably because the upgraded Apache's mod_ssl used a different version of the SSL library than the one mod_wsgi was built with. See https://github.com/GrahamDumpleton/mod_wsgi/issues/370. However, before this was diagnosed, I switched mpo to use Gunicorn with Apache.
The third server is Ubuntu 16.04 with Python 3.5, so it was OK as is.
For these installs, I opted to install the entire Mailman suite in a virtualenv. The biggest motivation for this choice was the fact that there are already production lists on both servers and using a virtualenv allowed me to do a lot of the work in advance without interfering with the production install. Possibly influenced by these experiences, I am now using virtualenv for all installs.
Note that all of this is done as the mailman user.
I already had some things set up in /opt/mailman including a git subdirectory containing clones of the GitLab mailman, mailmanclient, mailman-hyperkitty, hyperkitty, django-mailman3 and postorius projects. On the third server I have added mailman-suite because I use the settings.py from that project as the basis for mine.
I then created a /opt/mailman/mm directory and within that a Python 3.6 virtualenv named /opt/mailman/mm/venv. I then activated the virtualenv and did
pip install psycopg2-binary
pip install pylibmc
pip install Whoosh
on both servers. These are dependencies for using the PostgreSQL database, memcached and the Whoosh backend for HyperKitty archive search.
I also did the same, but in a Python 3.5 virtualenv on the third server.
Then because lm3o uses nginx and Gunicorn to support wsgi apps, I did
pip install gunicorn
there, and on mpo which uses Apache and mod_wsgi, I did
pip install mod_wsgi
but as noted above, mpo is now using Apache and Gunicorn like the third server.
The third server uses Apache and Gunicorn so I did
pip install gunicorn
The prior install on both servers used Mailman Bundler which is obsolete, however for consistency I kept some of the same names. Everything is in the /opt/mailman/ directory. I maintained the prior git/ subdirectory along with two bash scripts, pull and build which are used to pull changes from GitLab and build everything in the venv.
I maintained this on the third server as well.
Here are my recommendations for the rest of the configuration. The older installs don't adhere to this exactly, but this is what I'm doing going forward. Note: prior to 26 July 2020 there were paths in the files lm3o_nginx.txt, lm3o_nginx2.txt and mpo_wsgi.txt that were inconsistent with the layout below. These have been updated to be consistent.
/opt/mailman/mm/settings.py (copy of the mailman-suite settings.py file but with DEBUG = False)
if using Gunicorn:
and a symlink:
/opt/mailman/mm/logs -> /opt/mailman/mm/var/logs
In the /opt/mailman/mm/bin directory, create these executables:
/opt/mailman/mm/bin/django-admin (sample) Script to run Django management commands.
/opt/mailman/mm/bin/mailman (sample) Script to run mailman commands.
/opt/mailman/mm/bin/mailman-post-update (sample) Script to update static web and run migrations following a software update.
if using Gunicorn:
/opt/mailman/mm/bin/gunicorn (sample) Script to start Gunicorn.
There are several default templates that have incorrect or generic URLs for the web UI. To make these better, I installed these templates in /opt/mailman/mm/var/templates/site/en/ as follows:
These are the lm3o templates. The mpo templates are the same except for URL. Of course, if you use these you need to edit the URLs for your site.
I use Postfix and configuration is straightforward. Mailman's default configuration is appropriate for Postfix listening on 127.0.0.1:25. The main.cf settings are well documented at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#postfix. All that is really needed with recent postfix is
recipient_delimiter = +
and the additions to transport_maps, local_recipient_maps and relay_domains as described at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#transport-maps or, if you are using alias domains, the additions to transport_maps, virtual_alias_maps and relay_domains as described at https://mailman.readthedocs.io/en/latest/src/mailman/docs/mta.html#unusual-postfix-configuration.
As noted above the three servers I'm describing all have different web server configurations as follows:
nginx and Gunicorn
Apache and mod_wsgi
Here's an Apache configuration for mod_wsgi which I no longer use.
Apache and Gunicorn
Here's an Apache configuration for Gunicorn.
Finally, we need to arrange that all the services are running. Generally, the OS has taken care of init, upstart or systemd scripts for Postfix, the web server, PostgreSQL and memcached, but other things that need to be run are Mailman core, Django's qcluster and Gunicorn if that is used. Also, there are periodic crons that need to be run. Here are some samples.
Init mailman init.d.
Cron Mailman's crontab.
This was written assuming that the reader is familiar with Python software installation and tools like git and virtualenv.
Also, please note the first paragraph of the introduction. This is documentation of my decisions and experiences. It is not a how-to and does not replace the documentation at http://docs.list.org/
I've been asked to provide more information about some of the things that may be unfamiliar to some users.
In the Migration section, I refer to a /opt/mailman/git/ directory containing clones of the various GitLab projects and the 'pull' and 'build' scripts to pull and install updates. First, you need git which can be installed via your OS package manager if you don't have it. The cloned projects themselves are created in the /opt/mailman/git/ directory. This is done using one of two git commands for each project. If you have a gitlab account with your SSH key(s), you can
git clone firstname.lastname@example.org:mailman/project_name.git
for each project. Otherwise, you can
git clone https://gitlab.com/mailman/project_name.git
I also mention in the Installation section, creating a Python 3.6 virtualenv. Python 3.6 is currently the minimum required. It could just as well be any later Python 3 version and should be the latest available on your server.
Creating the /opt/mailman/mm/venv virtualenv is done by
or if you want to specify a Python other than the default
virtualenv -p /path/to/desired/python venv
This is then activated by
When active in a particular shell, it can be deactivated with the command
or by exiting that shell.
If you don't have virtualenv, it can be installed with pip or via your OS package manager. For the latter, it might be a package named virtualenv or python3-virtualenv.