SRE Offboarding

From Wikitech
(Redirected from Ops Offboarding)

Phabricator ticket

[] update LDAP permissions based on NDA status
[] update Phabricator permissions based on NDA status 
[] update [https://github.com/orgs/wikimedia/people github] access based on NDA status
[] Check HBase/Hadoop permissions and inform the SRE analytics team
[] update user in [[https://github.com/wikimedia/puppet/blob/production/modules/admin/data/data.yaml | modules/admin/data/data.yaml]]
[] run the logout cookbook
Additional task for SRE team members
[] Review access to internal IRC channels
[] Remove from ops mailing lists (ops and ops-private)
[] Remove from private Exim aliases
[] Remove VictorOps user
[] Remove Icinga user
[] Remove from pwstore
[] Review access to network devices (and potentially remove access)
[] Remove Kerberos principal (if present)

All Users

The following sections need to be completed for all users, regardless of team.

Volunteer status

Email the user and ask if they wish to remain on as a volunteer. If so, ensure they have signed an NDA.

Check user's LDAP access

When Offboarding users in LDAP, you have a number of options:

  • Remove the user from privileged groups, leave them in all unprivileged groups. This is the default and should be used in most cases.
  • Convert the user into a volunteer (requires a signed NDA). This happens in a few select cases.
  • Completely remove the user. This is for exceptional cases (rogue accounts, malicious users, etc).

The following commands can be run from mwmaint1002.eqiad.wmnet.

Remove user from privileged groups

This is the default action if a user does not want to become a volunteer. 

offboard-user -l $user

Convert user to a volunteer account

First ensure the user has signed the NDA. If so, run: 

offboard-user --turn-volunteer -l $user

Completely remove user

By default all non-privileged LDAP/Phab groups are retained! To completely remove a user, we can run: 

offboard-user --drop-all -l $user

Applying the change

These commands will create the LDIF file to use for removal and print the command you need to run. For example: 

 
$ cat jbond.ldif
dn: cn=ops,ou=groups,dc=wikimedia,dc=org 
changetype: modify 
delete: member 
member: uid=jbond,ou=people,dc=wikimedia,dc=org 

dn: cn=wmf,ou=groups,dc=wikimedia,dc=org 
changetype: modify 
delete: member 

member: uid=jbond,ou=people,dc=wikimedia,dc=org

$ ldapmodify -h ldap-labs.eqiad.wikimedia.org -p 389 -x -D "cn=scriptuser,ou=profile,dc=wikimedia,dc=org" -W -f jbond.ldif

If you just wish to check a users permission you can run the following command (we will use this for checking Hadoop permissions later): 

offboard-user --list-only -l $user

Check user's Phabricator access

If the user is remaining with Wikimedia as a volunteer, and their Phabricator account is linked to an LDAP account, or their Phabricator account is linked to a mediawiki.org account which does not end in "(WMF)", then this step can be skipped. Check their profile page in Phabricator to see what their Phab account is linked to.

Search for the username in Phabricator. Once you have the user's Phabricator username, run the following command (again, you can use --list-only to check first): 

offboard-user -p $user

If the user is a member of any privileged groups (e.g. Security, WMF-NDA) then remove them when prompted.

If the Phab account is connected only to a WMF SUL/OAuth account on mediawiki.org (see the user profile on Phabricator), then disable the Phab user account. This makes it clear that the person is not active anymore, and allows searching for tasks still assigned to them. In that case the user cannot log in to Phab anyway, as their SUL/OAuth account on mediawiki.org is also disabled.

Handle Hadoop leftovers

We first run the following script to check if the user has any Hadoop permissions: 

offboard-user --list-only -l $user

You need to make a list of all entries that mention Hadoop. In the following example, we need to make a list of the groups statistics-privatedata-users and analytics-privatedata-users 

User DN: uid=jbond,ou=people,dc=wikimedia,dc=org
Is member of the following unprivileged LDAP groups:
  cn=project-bastion,ou=groups,dc=wikimedia,dc=org (can be retained)
  cn=project-deployment-prep,ou=groups,dc=wikimedia,dc=org (can be retained)
  cn=project-tools,ou=groups,dc=wikimedia,dc=org (can be retained)
  cn=tools.perflogbot,ou=servicegroups,dc=wikimedia,dc=org (can be retained)
Is not a project admin in Nova
Is not a member in any privileged group
statistics-privatedata-users grants access to Hadoop/Hive, check PII leftovers
analytics-privatedata-users grants access to Hadoop/Hive, check PII leftovers

Once you have this information, create a new Phabricator task. Tag it with Analytics and link it to your own offboarding ticket. You can use the following for the ticket text: 

$Full_Name ($username) is currently being offboarded. Please check if they left anything in their home dirs on stat*/notebook*/HDFS since they were part of:

statistics-privatedata-users
analytics-privatedata-users

Validate shell access to the prod cluster

To remove a shell user, we will need to prepare a patch against modules/admin/data/data.yaml to perform the following tasks:

  • Add the user to the groups['absent'] at the very top of the file.
  • Remove the user from all other groups.
  • Update the users entry to:
 * Remove the email entry.
 * Blank out the ssh keys; i.e. set it to ssh-keys: []
 * Mark the user as absent.

Look at a real patch in Gerrit to see what this looks like.

Once this change has been accepted into Puppet, the user will be removed from all machines in ~30 minutes. It is possible that a user may still have a process running (e.g. screen, tmux). In these cases Puppet will fail, as userdel is unable to remove the user. We will get an alert for this, so monitor #wikimedia-operations for any alerts and intervene manually. If it is obvious the running process is not doing anything useful, then just kill it; otherwise, double-check with someone from the user's team.

Or, to get a list of running processes, you can get the user's uid from LDAP on the mwmaint host and run the following cumin command: 

$ sudo cumin '*' 'ps --user  $user_uid_number'

Run the logout cookbook

Once you have removed the user from admin's data.yaml, and Puppet has run everywhere, you should run the logout cookbook to force all user sessions to terminate: 

$ cookbook sre.idm.logout -u $user_uid -c $user_cn '*'

Remove user from Github

If the user does not want to remain as a volunteer, then we should also remove them from the Github org.

FIXME: This is just a placeholder; the author is not aware of the specific instructions for this.

People with higher privileges

Some of these changes require you to edit the private git repo. When doing so, ensure you have read the README file in /srv/private. Specifically:

  • DO NOT EVER rewrite history in this repo. No git commit --amend, no git rebase.
  • All changes, and the git commit, need to be performed as root or with sudo.

Review access to internal IRC channels

We need to ensure users are removed from private IRC chat channels. If you are not aware of the private channels, or their associated ops, please contact a member of the SRE security team. Specifically:

  • Review the ircservserv-config repo.
  • Remove from SRE private team channels, e.g. #wikimedia-sre-private.
  • Check with the user's team members to ensure they are removed from any team private rooms, e.g. #wikimedia-sre-traffic-private.

Remove from ops mailing lists (ops and ops-private)

First, you can check which lists a user is on with the following command on the mailman server (lists1001.wikimedia.org at time of writing): 

$ sudo mailman findmember user@example.org

To remove a user from all our mailing lists, run the following command: 

$ sudo mailman delmembers --fromall --member user@example.org

More information can be found on the Mailman page.

Remove from Exim aliases from private.git

Remove the user from all alias files under the following directory: 

/srv/private/modules/privateexim/files/

New users will probably only have one entry for root@wikimedia.org. However, users who have been around for some time may appear in multiple locations.

SRE Staff

For SRE staff, there are additional steps we need to perform.

Remove VictorOps user

Make sure the user has been deleted from https://portal.victorops.com/dash/wikimedia#/users.

Remove Icinga user

Create a patch to the Puppet repository to remove all instances of the user from modules/icinga/files/cgi.cfg. Note the username shown here is the LDAP (Wikitech wiki) username, which is not necessarily the same as the shell account.

Remove from pwstore

For this you will need to contact Jesse, Moritz, Riccardo or Simon and ask them to perform this task. Check the Pwstore page for more information.

Remove from Google group for private groups

Email Mark or Moritz and ask them to remove the user from the sre@wikimedia.org Google group.

Review access to network devices (and potentially remove access)

Please review whether the user had access to network devices at https://github.com/wikimedia/operations-homer-public/blob/master/config/common.yaml

If so, open a Phabricator task asking for network access to be removed and tag it with "netops"

Remove Kerberos principal

This is very simple, here is an example for user example_user@WIKIMEDIA: 

$ sudo manage_principals.py delete example_user@WIKIMEDIA

Also see the main page: Data_Engineering/Systems/Kerberos/Administration#Delete_a_principal_for_a_real_user

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=SRE_Offboarding&oldid=2165331"