Jump to content

SRE/Clinic Duty/Access requests

From Wikitech

LDAP access

Access to a range of mostly web-based services is granted via the "wmf" and "nda" groups. The specific permissions are listed at LDAP Groups. The change should be tracked in a ticket.

We're slowing migrating access requests for LDAP groups to Wikimedia IDM/Bitu. The current list of groups managed via Wikimedia IDM can be found at SRE/LDAP/Groups/Request access

Checklist

Users should create the phabricator ticket using the link on the following LDAP-Access-Requests project page. If that has not taken place then you need to ensure that the task has the LDAP-Access-Requests tag and also add the following checklist to the task description and make sure its completed before actioning the request 

[] The **username** of your existing account on wikitech.wikimedia.org: 
[] Do you currently have **shell access** (Yes/No)?
[] **Purpose** (Specify which service you need to get access to, e.g. Icinga, Grafana, Superset etc):
[] The specific **LDAP group** that you want to be added to (optional):
//For contractors only://
[] Contract end date:
[] Contract contact person:

WMF Group

The wmf group is no longer handled via Phabricator, instead access is being requested via Wikimedia IDM (https://idm.wikimedia.org). If you run into tasks when this is still requested, please add a message to point the user to request the group by following the steps listed at SRE/LDAP/Groups/Request access#Using the Wikimedia Identity Management System. Following that, the task can be closed.

NDA Group

  • For contractors that do not exist in LDAP (i.e. they do not have a wikimedia.org email address), ask for the person's manager/point of contact for approval, on the phabricator task
  • Volunteers and researchers can be added to the "nda" group, assuming they have signed an NDA which can be confirmed by adding @KFrancis to the Phab task.
  • Also add the user to the wmf-nda Phabricator group (click "add members", see phab:T299839 / T290605 for background on this).
  • Ensure a corresponding user entry exists in data.yaml (typically within ldap_only_users).

WMDE Group

Anyone at Wikimedia Deutschland who wants to get added to the "wmde" LDAP group needs to sign an NDA with the Legal department of the WMF. Simply add @KFrancis to the Phab task and she'll deal with it.

The access to "wmde" needs to be approved by an engineering manager from Wikimedia Deutschland. You can add either of the people below to the Phab task:

Finally, when granting access, ensure that the user is added to both ldap groups "nda" and "wmde"

Analytics Groups

  • There are multiple potential groups. They have been detailed on Analytics/Data_access#Access_Groups.
    • The clinic duty person can often link to this page for the person requesting access, and require the requestor to define which of the groups are required.
    • The clinic duty person should also highlight Analytics/Data access#User responsibilities to the requestor.
  • Make sure to seek signoff from Analytics folks on access tasks

analytics-privatedata-users

The analytics-privatedata-users Unix group is one of the more confusing groups as it can be configured in three different ways. Either: no ssh (no shell) and no Kerberos; no Kerberos (standard admin.yaml) or as a shell account with Kerberos. In all cases the user should be configured under the user section of admin.yaml and not the ldap_only_user section.

Data Engineering/Systems/Kerberos/Administration#Create a principal for a real user describes the steps how to add a Kerberos user principal for a user

Please tag the access request ticket with "Data-Engineering" to increase visibility.

Tickets asking for membership for non-WMF & non-WMDE staff in analytics-privatedata-users need approval from one of the group owners listed in data.yaml. WMF or WMDE staff do not need explicit approval from group owners[1][2].

Other groups

  • All other groups need to be approved by the user's manager

Create/Get LDAP account

In order to add or update a a user's LDAP permissions, they will first need to create a developer account. In either case you will need to know the username used. You can also search ldap to try and find it: 

ldap-maint1001$ ldapsearch -x mail=user@example.org

If the user claims to be a Wikimedia Foundation staff member or contractor, you should verify that by following the instructions in #Verifying WMF developer accounts.

Update data.yaml

Check whether there's an existing entry in modules/admin/data/data.yaml:

  • If the user already has shell access, no further change is needed. You can proceed with the LDAP change below:
  • If not, add the user to the ldap_only_users table at the end of the file, using their would-be shell username (LDAP uid=) as the key within ldap_only_users. (This is just added for tracking/verification purposes. As you'll be making the LDAP changes yourself, no puppet run is required after this.)
    • Add the realname of the user (most Cloud VPS accounts don't have a real name set)
    • Add the email address of the users:
      • If the user is WMF staff, use the email address of their Google account (You can double-check the account name in the Gmail interface). Some users have aliases for their nickname; don't use these, use the official Google account (this allows cross-checking data against corp LDAP)
      • If the user is not staff, ask for a contact email address (to have a reliable contact e.g. in case of an account compromise)
    • If the user is someone with a time-limited access e.g. interns, researchers who have time-limited MOUs or short term contractor, add the estimated account end date as expiry_date (format is YYYY-MM-DD) and a staff contact as expiry_contact

The entry should look something like the following:

exampleuser:
  ensure: present
  realname: Example User
  email: exampleuser@example.org
  expiry_date: 2038-01-19 
  expiry_contact: examplestaff@wikimedia.org

Modify LDAP groups

After having added the user to data.yaml, the change in LDAP can be done from one of the LDAP maintenance hosts like ldap-maint1001 (this will be automated in a subsequent step):

  • Check if they are a member of the group from the Cloud VPS LDAP server: ldapsearch -x cn=grpname
  • Add them if they are not there: sudo modify-ldap-group grpname and add their entry in the editor window that pops up
  • To remove someone from an ldap group you can sudo modify-ldap-group grpname and delete their entry in the editor window that pops up
If a user has to be removed from special LDAP access, in most cases (e.g. contract termination) you may want to notify also @aklapper to remove/check Phabricator access on the same ticket.

For further instructions see Help:Accessing Cloud VPS instances, LDAP and LDAP Groups.

Production shell access

Access and reasoning for requesting it are documented on Requesting shell access. Please read and understand entirely before processing any access requests, as this very brief summary documentation may not cover all required points in the linked page.

If a request asks for things like new shell accounts, access to additional servers, log files, personal data, admin roles in systems like Mailman, Bugzilla, data center access, opening a firewall rule etc, then it is an access request and should be moved into the SRE-Access-Requests Project.

Check list

Users should create requests using the the access request form, if this has not been followed then you will need to make sure the following checklist is copy/pasted to the task and completed before processing the request. 

[] - User has signed the L3 Acknowledgement of Wikimedia Server Access Responsibilities Document.
[] - User has a valid NDA on file with WMF legal.  (All WMF Staff/Contractor hiring are covered by NDA.  Other users can be validated via the NDA tracking sheet)
[] - User has provided the following: wikitech username, email address, and full reasoning for access (including what commands and/or tasks they expect to perform)
[] - User has provided a public SSH key.  This ssh key pair should only be used for WMF cluster access, and not shared with any other service (this includes not sharing with WMCS access, no shared keys.)
[] - access request (or expansion) has sign off of WMF sponsor/manager (sponsor for volunteers, manager for wmf staff)
[] - access request (or expansion) has sign off of group approver indicated by the approval field in data.yaml

More details on the above:

  • The L3 acknowledgement signature list.
  • User's direct supervisor has approved of access request via comment on phabricator task.
  • Approval from project lead where user's access will be granted via comment on phabricator task. See above for Staff exceptions.
  • Confirmation that the user has read, comprehend, and signed the Acknowledgement of Wikimedia Server Access Responsibilities document.
  • ALL ACCESS REQUESTS REQUIRE AN NDA.
    • Anyone who's Wikimedia staff has signed an NDA as part of their work contract. To validate that someone is staff you can either (1) check ldap-corp1001 via an LDAP search for the GCorp username (2) have the person's manager confirm (they need to sign off anyway)
    • Volunteers and researchers need to sign an NDA with the legal department. You can check existing NDAs on file at https://docs.google.com/spreadsheets/d/1xQNx5s2yErvayCMzvk9VkIA2ZihFXSBEhT5Z5ziCsi4
    • If that's not the case, add @KFrancis to the Phabricator task to prepare an NDA (she'll confirm on task when that's completed)
  • SSH public key has to be submitted via gerrit patchset by user, or by some confirmed (non-email) method (suggestion: wiki user page).
    • If a group is marked as require_fido: true, then all of the user's SSH key(s) must be FIDO-backed keys (see Yubikey-SSH-FIDO)
    • You can verify that the SSH key is not used in WMCS by running from the current LDAP maintenance host (ldap-maint1001 as of July 2025) the cross-validate-accounts command, for example like:
      cross-validate-accounts --username USRNAME --uid 00000 --email address@example.com --real-name "Real User" --ssh-key "ssh-ed25519 AAAAC...." --kerberos
  • Any requests to add sudo rights to a group should be reviewed at the Infrastructure Foundations SRE meeting. Tag the task with #Infrastructure-Foundations and assign it to @LSobanski to add it to the next meeting's agenda; don't grant access until it's discussed at the meeting. Skip this step for requests to add a user to an existing group (the most common kind of production access request).
  • Please update the Task in phabricator, as the requestor will get update.
  • Please raise any security concerns on ticket via comments.

Deployment Groups

  • Requires Shell and must have approval from releng to be added to the deployment group

Creating new shell users

Please see instructions in the puppet admin module's README.

Some notable changes since February 2017:

  • Add the realname of the user (most Cloud VPS accounts don't have a real name set)
  • Add the email address of the users:
    • If the user is WMF staff use the email address of their Google account (usually the first letter of the first name and the surname, you can double-check the account name in the Gmail interface). Some users have aliases for their nickname e.g., don't use these, use the official Google account (this allows cross-checking data against corp LDAP)
    • If the user is a volunteer, a researcher or contractor without access to a wikimedia.org account, ask for a contact email address (to have a reliable contact e.g. in case of an account compromise)
  • If the user to be added is someone with a time-limited access (e.g. interns, researchers (who have time-limited MOUs) or short term contractor), add the estimated account end date as expiry_date (format is YYYY-MM-DD) and add a staff contact as expiry_contact

Renaming shell users

Sometimes we have to rename a shell user. This is typically when their shell name doesn't match their login name, and they have issues logging into items requiring LDAP credentials.

Renaming a user will require a few things happen, in a very specific order. Since many users keep data in their home directories, backups can sometimes be made, but not always. (Private data that isn't allowed to be copied off the cluster should not be backed up to laptops.) The existing username has to be removed from the host, since the new username will use the old username's UID.

  • Patchset is prepared, but not merged.
  • Using cumin for these batch commands, all hosts that have the existing (to be replaced) username should have puppet halted.
  • Affected hosts should have the user (to be replaced) deleted. DO NOT DELETE THE USER'S HOME DIRECTORY.
  • Merge patchset with username change (UID remains the same).
  • Run puppet on affected hosts, and they will create the new user (using the same UID.)
  • Batch move the contents of the old user home into the new user home.

IRC channel access

/query chanserv
help access
access #channel list
access #channel add *!*@wikimedia/cloak 

14:07 -ChanServ(ChanServ@services.)- Flags +Aiortv were set on ...

For people wanting to be a channel operator for #wikimedia-operations, first check they got nick protection enabled 

  /msg nickserv info <nick>
  ...
  <nick> has enabled nick protection

and then 

  /msg chanserv flags #wikimedia-operations <nick> +Aiotv

Check on a Phabricator user

As part of an access request you might want to check first if a Phabricator user is actually who they say they are. There is a shell command on the the Phabricator server. see Phabricator#Check_on_a_Phabricator_user

Search console

Access to Google Search Console and Bing Webmaster Tools is no longer owned by SREs. IT Services owns it, approving and deploying the access as needed. Normally, they will be able to self serve and no work from us will be needed. However, we still have a small role overseeing and assisting on those requests, upon their request:

  • We still keep admin passwords in pwstore in case the current administrator in unavailable or their account gets disabled
  • We manage the external service authentication through DNS keys (in case new services or domains have to be added. e.g.).

The following instructions are kept just in case:

Google Search Console

  • There is no scripted check for expiry at this time. Add the user to the maint-announce calendar for their expiry until we have a better system.

Adding users to domains

  • This is a tedious process. Once you login to the Google Search Console you will see a list of every single sub-domain for every single language and project in both HTTP and HTTPS.
  • Login from an incognito browser window or a profile where you're not already logged in with any other Google account to google search console using the admin account in pwstore. This requires a verification code that will be sent via email.
  • Select the domain you wish to manage from the search/drop down in the top left of the page
  • Select settings from the menu, then User and Permissions
  • In almost all cases, the user should only be given Restricted to the Domain.
  • Click add user, and then enter the user's email address and click add user. Permission should be 'Restricted'.
  • User has now been added to that project/subdomain. Repeat as needed for other projects/subdomains.
  • For NON-WMF ADDITIONS: An entry should be added to the maint-announce calendar, set to email out to the root list and the person who approved the access to the search console for the entry to alert them of expiry. Reach out to the SRE Clinic on-duty to request this. Request SCherukuwada@ to remove the user from each domain.

Removing users from Domains

  • Login from an incognito browser window or a profile where you're not already logged in with any other Google account to google search console.
  • Select the domain you wish to manage from the search/drop down in the top left of the page
  • Click on Manage Domain, Add or Manage Users for the single domain you want to edit. This pulls up a sub-page.
  • Click the user in question in the list presented on page, this highlights the user.
  • Click the delete user button on the highlighted line.

Bing Webmaster Tools

  • There is no scripted check for expiry at this time. Add user to the maint-announce calendar for their expiry until we have a better system.

Adding Users to Domains

  • Login from an incognito browser window or a profile where you're not already logged in with any other Google account to Bing Webmaster Tools using the admin account in pwstore.
  • Once you login to Webmaster Tools you will see a menu item called "User Management" on the left sidebar.
  • Bing has been configured with all the wikipedia.org domains added together. There is no way to grant access on a per-language-domain basis as of this writing.
  • Click on "User Management" then on the "Add a User" button and give them "Read Only" access.

IMPORTANT NOTE FOR NON-WMF ADDITIONS: As with Google Search console, we need to ensure that the granted permission has an expiration date. An entry should be added to the maint-announce calendar, set to email out to the root list and the person who approved the access to the search console for the entry to alert them of expiry. Then whoever is on clinic duty for that week needs to login and remove the user from each of the domains.

Removing users from domains

  • Login from an incognito browser window or a profile where you're not already logged in with any other Google account to Bing Webmaster Tools.
  • Click on "User Management".
  • Click on the triple-dot menu alongside the user you wish to remove and then click on "Remove User".

Verifying WMF developer accounts

Both LDAP and shell access is tied to a user's developer account, so a key task is verifying that a developer account actually belongs to the claimed person.

You can log into ldap-maint1001.eqiad.wmnet and run the following ldapsearch variants: 

# Searching for the Wikimedia Developer account name:
$ ldapsearch -x cn=foo

# Searching for the shell/UID:
$ ldapsearch -x uid=foo

# Searching for the email address:
$ ldapsearch -x mail=foo@bar.org

WMF staff shows up with their @wikimedia.org email address.

If the user has linked their Wikimedia Developer account to a SUL account, it will show up under the wikimediaGlobalAccountName attribute. All WMF staff members will have a SUL user name with (WMF) as a suffix.

  1. ↑ T370424 Streamline Data Platform access approvals for WMF staff
  2. ↑ T381824 Data Platform access streamlining for WMDE staff
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=SRE/Clinic_Duty/Access_requests&oldid=2376242"