Mailman/Mailman2

This page contains historical information. We no longer use Mailman2

How to

Create a mailing list

There are 2 ways to create a mailing list:

Via the web interface at http://lists.wikimedia.org/mailman/create - a list's creator password is needed. The site password works as well.
Via shell on the mailing lists server (lists1001.wikimedia.org). As root, run newlist.

In both cases, it's not necessary to add e-mail aliases anywhere! These are handled automatically by mailman with the mail transfer agent configured (exim). When creating a new mailing list, make sure it follows the Naming scheme.

Step-by-step procedure using the UI

Have a phabricator task with the following info: Requested name, Initial list administrator's email address, Secondary list administrator's email address, description.
Have the list creator or master passwords.
Go to https://lists.wikimedia.org/mailman/create
Fill the "Name of list" and "Initial list owner address" fields and set the "Auto-generate initial list password?" field to Yes
Click on "Create List". This will take you to a page with 2 links, the info link and the admin link. Copy/Paste them on a note pad
Follow the admin link (https://lists.wikimedia.org/mailman/admin/<list-name>). Use the master password to enter. Leave everything as is except:
- Check the public name of the list and eventually adjust its case
- Add second admin
- Description
- Select Privacy settings and update 'What steps are required for subscription' to 'Require approval'
Reply to ticket with info and admin link, let the requester know that subscriptions have been set to require approval, however, all other options have been left with their default settings
Add the list to m:Mailing lists/Overview, or tell the requester to.

Note: Sample Phabricator ticket T207283

Disable or re-enable a mailing list

Sometimes a mailing list may be requested to be removed but not deleted from the server because the archives may still be used such as old announcement lists. Disabling and re-enabling a list can be done through the mailman interface by list administrators but usually it is easier for someone who is experienced to handle it due to the complexity of mailman configuration.

A shell script exists on the lists server under /usr/local/sbin/ called disable_list. The script can be used to disable and enable lists easily and consistently.

# Usage: ./disable_list [-e|--enable] <listname>

Real world example (from T200733)

[lists1001:~] $ sudo disable_list performance
/var/lib/mailman/data/heldmsg-performance-207.pck
/var/lib/mailman/data/heldmsg-performance-203.pck
/var/lib/mailman/data/heldmsg-performance-201.pck
/var/lib/mailman/data/heldmsg-performance-205.pck
/var/lib/mailman/data/heldmsg-performance-204.pck
/var/lib/mailman/data/heldmsg-performance-202.pck
/var/lib/mailman/data/heldmsg-performance-200.pck
/var/lib/mailman/data/heldmsg-performance-206.pck
performance disabled. Archives should be available at current location, all mail should be moderated and the list should not be on the listinfo page.
[lists1001:~] $

Commands

The shell script simply does the following to disable a list:

# list=LISTNAME
# echo "advertised=0" | config_list -i /dev/stdin $list
# echo "emergency=1"  | config_list -i /dev/stdin $list
# echo "member_moderation_action=2" | config_list -i /dev/stdin $list
# echo "generic_nonmember_action=2" | config_list -i /dev/stdin $list
# echo "ban_list=[^.*@.*]" | config_list -i /dev/stdin $list
# rm /var/lib/mailman/data/heldmsg-$list-*.pck

And to re-enable a list, it does the opposite of the above.

# list=LISTNAME
# echo "advertised=1" | config_list -i /dev/stdin $list
# echo "emergency=0"  | config_list -i /dev/stdin $list
# echo "member_moderation_action=0" | config_list -i /dev/stdin $list
# echo "generic_nonmember_action=1" | config_list -i /dev/stdin $list
# echo "ban_list=[]" | config_list -i /dev/stdin $list

After using this script it's nice if you login one last time on the listinfo page and remove the former admins from the "list run by" field. This makes sure they never get any spam to the list owner alias. When you login you will see a message that "emergency moderation" is enabled. This tells you the script did the job and the list is disabled.

Remove a mailing list

Removing a mailing list should only be done if the list will not be used at all and archives don't need to be public (otherwise see above). To remove a list but keep the archives stored, run (on the lists server):

# rmlist listname

To also remove all archives, use:

# rmlist -a listname

Remove a message from the mailing list archives

Sometimes you are forced to remove a message from mailing list archive by court order or other 'command' (such as request from legal). The linked instructions tell why it is a procedure best avoided but also how to do it if legally obligated.

Wikimedia's mailing list archives, though public, are no longer indexed by search engines as they are excluded in robots.txt, other archives (such as Gmane) are indexed.

In mailman3, log in with the root account (credentials in pwstore), and there will be a "Delete this message" button in hyperkitty if you're looking at the specific post.

Export a listing of all subscribers to a mailing list

Login to the mailing list server and run:

  /var/lib/mailman/bin/list_members -f -o <file to write to> <list name>

In Mailman3 run:

$ sudo mailman-wrapper members <listname>@lists.wikimedia.org

Check if an email address is subscribed to any lists

  [lists1001:~] $ sudo /var/lib/mailman/bin/find_member <email address>

In Mailman3 run:

$ sudo remove_from_lists --dry-run <email>

Remove an individual from all mailing lists

Occasionally we need to remove an individual from every mailing list we have, such as when an email address no longer works but we don't want mailman to turn it off due to bounce detection. The remove_members command is the solution - this is a command line utility to remove one or more email addresses from a specific list or from all lists.

remove an individual from a specific list:

 /var/lib/mailman/bin/remove_members mylist user@example.com

remove two addresses from all lists:

 /var/lib/mailman/bin/remove_members --nouserack --fromall user1@example.com user2@example.com

remove from all private lists:

 /var/lib/mailman/bin/remove_from_private.sh <email>

In Mailman3 you can do:

$ sudo remove_from_lists <email>
$ sudo remove_from_lists <email> --only-private

Reset the admin password of a list

TLDR

ssh lists1001.wikimedia.org
list_name="engineering" ; sudo /var/lib/mailman/bin/change_pw -l $list_name -p $(pwgen -c1 -s 12)

(replace "engineering" with the list name)

details

If there is a request to reset a list password, first go to the admin page of the list, example if the list is called "wikimedia-l", the URL is: https://lists.wikimedia.org/mailman/admin/wikimedia-l
Check who is listed in the "list run by" sentence in the footer. Note how when you hover over the admin names there is a mailto: link with a special list owner alias, example for wikimedia-l it's wikimedia-l-owner@lists.wikimedia.org. Use that address to contact list admins, not random personal emails. It's the best way to ensure the request for a reset is valid and ALL admins get the new password, not just one of them, causing follow-up requests. Realize that there is just a single admin password all admins have to share and keep in sync.
If it appears the request can be handled by one of the existing list admins, for example there are multiple admins and not all of them have stated they don't know the passwords, let them handle it and decentralize admin work where possible. Only if really needed, proceed to:
Get the mailman master password from pwstore
Login on the admin page using the master password and go to the "Passwords" section.
Fill out the "Enter new administrator password:" form with a new random password and repeat it. There are also poster and moderator passwords but they are almost never used and if they are then let the admins handle giving out passwords to moderators. Click "Submit your changes"
Send the new password to the list owner address described above. If a phabricator ticket exists for this (recommended!) and list admins are on phabricator, add them to ticket and resolve.

OR (alternative way to do it)

ssh to the list server (currently lists1001.wikimedia.org)
cd /var/lib/mailman/bin/
./change_pw --help
./change_pw -l <listname> -p <newpassword>
This should automatically mail the list owner address because you did not use the -q (quiet) option. So if the people listed as admins are current everything should be done.

Rename a mailing list

Renaming a mailing list in the past was a fairly hands-on approach and required confidence with mailman at each stage. While this has not changed, a shell script has been made which makes the process more easier and more streamlined and reduces the risk of things going wrong within mailman itself. Although this is true, renames should be avoided without someone who is confident with mailman on-hand just in case any sort of issues pop up. Tailing the error and bounce logs as well before and after would be ideal.

The script can be ran on the lists server by running:

/usr/local/sbin/rename_list <old list> <new list>

The password and email address of the list can be set to anything (preferable not the list administrator's address) as it will be overwritten later on by the script.

Manual

The manual process is documented below (and may be out of date as new optimisations are found) which can be done if the script does not work or the list is 'weird' in one way or another. It is best to avoid the manual method as it introduces more risks and human errors.

Before you begin, verify that the requester is actually an administrator of the mailman list.
Send an email to the -owner address of the list that you plan to rename a list.
Create a new list.
1. Use the mailman master password as creator's (authentication) password.
2. Follow the standardized naming scheme where possible.
3. Use your email as the initial list creator and receive auto-created password (will be overwritten by the move).
4. Log into the administrative interface (https://lists.wikimedia.org/mailman/admin/<listname>) with the generated password or the site password.
Before the next step be prepared to change the "real_name" value of the list in the interface, but don't send it yet. Have the mailman master password ready.
Go to the lists server shell and copy the config.pck (this includes all settings, users, passwords!) and others (pending.pck, request.pck files) from old to the new list.
Reload the administrative interface link and be logged out, because you have just overwritten the users and passwords as well, use the mailman master password and login again.
Immediately change the real_name in the administrative interface.
You should now see other users as list administrators and members, and now you can take your time and adjust other settings like the description field and "Prefix for subject line of list postings" and update the name there as well or let others do it.
Copy the archive mbox from the old to new .mbox directory and rename it to reflect the new list name.
Fix permissions (chown list:list *.mbox, chmod 664 *.mbox)
Use arch to recreate all html files from mbox to fix archive links.
Decide if you want to keep old archives in place, you probably do. Don't break URLs.
Add the old list email address to "acceptable aliases" on the new list administrative interface
Merge a mail alias to redirect mail to the old list.
Merge an url redirect for the old listinfo page.
Merge on palladium, run puppet on the mailing lists server. Apache and exim should automatically pick the change up.
Test URL redirect.
Test mail by announcing the change to the old list address.

Alter arbcom-l archive access list

This page contains historical information. It may be outdated or unreliable.

After a previous data leak, the English Wikipedia Arbitration Committee requested a high level of security for access to arbcom-l archives. We now have a second layer of password authentication in addition to Mailman's controls, implemented in Apache. Using HTTP authentication allows each user's page views to be tracked in the apache access logs.

To add a user:

Have the user generate a GPG private key, and have them establish a link between their public key and their Wikipedia user account by posting the public key on Wikipedia. For Windows users, this can be done by following this tutorial.
Generate a password for them, for example using pwgen 24
Choose a username.
Update the htdigest file in the puppet private repo using htdigest /srv/private/modules/secret/secrets/mailman/arbcom-l.htdigest 'arbcom-l archive' <username>
Push the update out to the mailing lists server in the usual way.
Send the new password to the user using encrypted email.

To do other things:

Edit puppetmaster1001:/srv/private/modules/secret/secrets/mailman/arbcom-l.htdigest, do a commit and run puppet on the mailing list servers (lists1001.wikimedia.org atm)

Configuration details

Mailman setup

The local mailman configuration file is available to see on Diffusion. Configuration is mostly default version config with few changes (L96 and below are non-default).