Mailman3 has multiple components. Each doing a different thing. This is a simplified overview. For a more visual diagram, see the upstream architecture overview.
- mailman3 is the service that handles emails and talks to MTA (in our case exim4) through LMTP/SMTP
- mailman3-web or postorius is a django-based web service that handles settings, subscriptions, etc. It doesn't do anything on its own (except user management) but it talks internally to the mailman3 service through REST API.
- hyperkitty (being used in the same webservice) is responsible for archives. It handles search, etc.
- Search backend: Is xapian and the search index is in /var/lib/mailman3/web/fulltext_xapian_index
- Databases: Two databases both in m5:
- mailman3 which mailman's core service talks to. It holds memberships, lists, etc.
- mailman3web which postorius and hyperkitty talk to. It holds archive of all emails and is rather big.
- MTA: It's exim4 (see Mailman/MTA for more information)
Who has access?
The following people have "superuser" access via the web UI for admin access to all lists:
- Quiddity (Nick Wilson) (WMF)
- "Administrator" (all SRE members or users with root)
- "WMFOffice" (Trust and Safety team)
Create a mailing list
- Go to https://lists.wikimedia.org/postorius/lists/new/ (as an administrator)
- or ssh to lists1001.wikimedia.org and run "sudo mailman-wrapper create --owner OWNER LISTNAME@lists.wikimedia.org"
Consider if you need to notify Trust and Safety when creating new mailing lists.
The only part of new mailing lists that can't be easily changed is its name. Pleas make sure it follows naming standards before creating.
Disable or re-enable a mailing list
- Remove owners and add firstname.lastname@example.org as owner
- This is important, otherwise we can't find it and change/fix its settings.
- Set all emails to be discarded (both member and non-member)
- Add to description that this mailing list is disabled like [Disabled] or [Archived]
Remove a mailing list
Note: unless you know what you're doing, don't actually do this. It most likely does not do what you want. Just disable the list (see above) instead.
- Go to https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/delete (as an administrator)
- or ssh to lists1001.wikimedia.org and run "sudo mailman-wrapper remove LISTNAME@lists.wikimedia.org"
Remove a message from the mailing list archives
- Delete the whole thread
- Go to the thread in hyperkitty (example).
- Click on "Delete this thread"
- Delete a single message
- Go to the message in hyperkitty (this can be tricky, go to the thread, click on the chain icon aka "Permanant link to this email"). example
- Click on "Delete this message"
Export a listing of all subscribers to a mailing list
- Go to members settings: https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/members/member/
- Click on "CSV export"
Check if an email address is subscribed to any lists
Remove an individual from all mailing lists
- ssh to lists1001.wikimedia.org
- Run "sudo mailman-wrapper delmembers -m EMAIL_ADDRESS --fromall"
Reset a user's password
- ssh to lists1001.wikimedia.org.
- Run "sudo mailman-web changepassword USERNAME"
Rename a mailing list
Import from Google Groups
It is straightforward to migrate a list from Google Groups to Mailman. A list owner in Google should use Google Takeout (see documentation) to export the member list (members.csv) and archives (topics.mbox). File a task requesting a new list and include a link to the Takeout result (if the list contents are private, provide this information to the Mailman admins/SRE team privately).
Create the new list in Mailman like normal, then use the "Mass subscription" form at <https://lists.wikimedia.org/postorius/lists/LISTNAME.lists.wikimedia.org/mass_subscribe/> to import the existing subscribers.
Copy the mbox to the mailing list server and run:
sudo mailman-web hyperkitty_import -l LISTNAME@lists.wikimedia.org /path/to/topics.mbox
This might take a while because of the "Synchronizing properties with Mailman" step, that's fine.
Then update the search index:
sudo mailman-web update_index_one_list LISTNAME@lists.wikimedia.org
All done! Make sure to have the list owner in Google disable the group there, and remind subscribers to update the lists' email address to LISTNAME@lists.wikimedia.org.
Add a list to Gmane
A new list can be added to Gmane very easily if the first message has not already been sent or history is not important. If the history needs to be imported, a ticket should be file in Phabricator with projects 'Wikimedia-Mailing-list' and 'operations' (easy link stating which list needs to imported and to which Gmane group if it exists already.
Authorized recipients for email@example.com
See Mailman/Migration for comments made about past migrations (lily to sodium, sodium to fermium, fermium to lists1001).
When upgrading Mailman3 packages for a version that includes schema changes, use the following process.
- Downtime host in icinga
- Disable puppet
- Manually stop
- Run apt to install the new packages
sudo mailman-wrapper helpto apply the mailman3 schema updates (yes, running just the "help" command will apply the schema changes)
sudo mailman-wrapper helpagain to make sure the schema update didn't obviously break anything.
sudo mailman-web migrateto apply the mailman3 schema updates
- Clear static CSS/JS caches:
sudo mailman-web collectstatic --clear --noinput && sudo mailman-web compress
- Start mailman3 and mailman3-web systemd services again, keep an eye on errors in the logs
- Re-enable puppet and remove icinga downtime if it hasn't expired yet.
- Verify that the messages the MediaWiki Installer screen-scrapes haven't changed.
Block abusive traffic
profile::lists::web_deny_conditionsin private puppet to block at the Apache level (see gerrit:736211)
- Use the standard
abuse_networksin private puppet which will block at the ferm/network level (see gerrit:736552)
The new Mailman3 setup lives on lists1001. Some state is under /var/lib/mailman3/, the global configuration is in /etc/mailman/ via Puppet.
We are currently using custom backported packages of the mailman3 suite. A few patches have been cherry-picked from upstream to fix issues we've run into. The packaging Git repositories can be found on apt1001 in legoktm's home directory.
There is currently one change that is hotpatched and not in the package yet.
Mail server setup
Web server setup
To get Mailman running with Apache the puppet class role::lists is applied which uses module mailman. Puppet module mailman uses puppet module Apache which sets up Apache and the site config is in the template lists.wikimedia.org.erb.
lists1001 is backed up to backup1001 using Bacula. All of /var/lib/mailman3 is backed up, which potentially includes on-disk templates and the search index. We do not back up /var/lib/mailman3/web which comes from the package and digests that haven't been sent yet.
The database lives on m5, which is also regularly backed up into Bacula, too.
Tested failure modes
Because mail delivery and transport should be reliable, I have tested what happens in certain failure modes, e.g. when SpamAssassin's spamd daemon is not running.
Spamd not running
Because of the /defer_ok modifiers in the Exim ACLs, Exim will act as if no spam filtering attempts are made when spamd is not running, and will accept the message. The following lines are logged:
spam acl condition: warning - spamd connection to 127.0.0.1, port 783 failed: Connection refused spam acl condition: all spamd servers failed H=xxx.xxxxxxx.xx [xx.xx.xx.xx]:xxxx I=[188.8.131.52]:25 U=exim Warning: ACL "warn" statement skipped: condition test deferred
Mailman not running
If the Mailman queue runner daemons are not running, incoming messages will still get delivered to the Mailman queue by Exim. However, nothing else will happen until the Mailman processes are started.
Most emails sent out by Mailman3 can be customized using "templates". You can see the default English templates. You can edit them by visiting https://lists.wikimedia.org/postorius/lists/
YOURLISTNAME.lists.wikimedia.org/templates. These are translatable on translatewiki.net.
It may also be useful to read the template documentation (the code parts can be ignored).