Jump to content

SRE/LDAP

From Wikitech
< SRE

Wikimedia's LDAP configurations apply to logins in Toolforge and Cloud VPSes using sssd. However, some configurations come directly from Puppet's data.yaml.

LDAP in Production

There are two separate LDAP directory trees running in production:

  • The main cluster is used for authentication to various developer-oriented services (including Toolforge, Cloud VPS and all services behind CAS-SSO). Accounts on this LDAP tree are called "Wikimedia developer accounts". This cluster has multiple endpoints:
    • Writes are sent to a pair of writable servers (one in each main DC, currently seaborgium/serpens). This endpoint is known as 'ldap-rw' (previously 'ldap-labs'), and should only be used by clients that need to change LDAP data.
    • Everything else should use the 'ldap-ro' endpoint (backed by multiple ldap-replica* hosts via LVS). These servers have a full replica of the LDAP tree and only exist to increase availability and decrease load from the writable servers.
  • cloudservices2XXX-dev hosts run a small cluster that is used like the main cluster on the OpenStack testing environment.

All installations are using OpenLDAP. The writable servers of each cluster are running in mirror_mode, which provides bi-directional replication of write changes using the syncrepl protocol.

Installing/Configuring the server

Create indexes

After creating the indexes, on each server you need to stop the server, rebuild the indexes, and start the server:

Add a proxy agent

Since we are requiring authentication by default, the clients need a user to do authenticated lookups. We create a proxyagent user for this: 

dn: cn=proxyagent,ou=profile,<basedn>
changetype: add
objectclass: top
objectclass: inetorgperson
objectclass: person
sn: agent
givenName: proxy
userpassword: <aPasswordGoesHere>
cn: proxyagent

Troubleshooting

Where to find logs

The main daemon logs can be found here: 

user@seaborgium:~$ sudo journalctl -u slapd.service

Other interesting logs are:

  • TODO: fill me

Corrupt nscd cache

The nscd cache is a bdb database, and can (rarely) become corrupt when a system shuts down uncleanly. You can forcibly clear the cache by doing the following: 

/etc/init.d/nscd stop
rm -f /var/cache/nscd/*
/etc/init.d/nscd start

Invalid nscd cache

To purge an invalid cache, you can do the following: 

nscd -i <database>

Where <database> is passwd, group, or services.

Debugging sudo-ldap

Sudo will print ldap debugging information, if you add the following to /etc/ldap/ldap.conf: 

SUDOERS_DEBUG 2

Debugging autofs

Autofs will print debugging information to /var/log/syslog, if you add the following to /etc/default/autofs: 

LOGGING="debug"

Common LDAP administrative actions

This shows you how to do user administration on LDAP. You need access to mwmaint1002 to perform write actions, and should be either a root or a member of the group ldap-admins. We shall use the wonderful ldapvi program, not invented by us but maintained by a real upstream, to do these tasks.

Display extended info about a user

[mwmaint1002:~] $ ldapsearch -x uid=$username

Replace $username with the actual user name (uid). You will see the "cn", "sn" and "uid". Note how these can be different. This also displays the email address and SSH key used. Useful to verify if a user has a WMF email address and doesn't have the same key as in production.

Show group members / check a user is already a member of a group

[mwmaint1002:~] $ ldapsearch -x cn=$grpname

Replace $grpname with the actual group name (e.g. wmf, ops, ...).

Find a user by email address

[mwmaint1002:~] $ ldapsearch -x mail="foobar@wikimedia.org"

[mwmaint1002:~] $ ldapsearch -x mail="ffoobar*"

This works with an exact email address or wildcards as well. Useful to find a user if the UID they picked is unknown but you know they used their (WMF) email address which can be derived from first and last name.

Add a user to a group

Method 1

[mwmaint1002:~] $ sudo modify-ldap-group wmf ('wmf' is the group name, other valid groups are 'ops', 'wmf', 'nda' or 'wmde')

An editor will open, copy/paste/modify a line, write and quit.

  • When adding a user to the wmf LDAP group, please also add them to the Phabricator group wmf-nda (click 'members'->'add members'), see T290605 for background on this).

Method 2

ldapvi -b ou=groups cn=$groupname

This brings up an editor with all information about the group, including its list of members. You can add a member by adding the following line to the end of the editor:

member: uid=$shellname,ou=people,dc=wikimedia,dc=org

Then save and exit your editor (<ESC>:wq in vim), you'll be asked if you want to save these LDAP changes. Press y and enter to confirm, or q and enter to cancel

Remove a user from a group

Method 1

[mwmaint1002:~] $ sudo modify-ldap-group wmf ('wmf' is the group name, other valid groups are 'ops', 'wmf', 'nda' or 'wmde')

An editor will open, delete the relevant line, write and quit.

Method 2

ldapvi -b ou=groups cn=$groupname

This brings up an editor with all information about the group, including its list of members. You can find the member you want to remove (Press /$shellname to search in vim), remove that line entirely, and then exit and save.

Finding the shell name of a user

ldapsearch -x -b ou=people,dc=wikimedia,dc=org cn=$wikitechUserName

The shell username will be listed as 'uid', which is also in the DN.

Editing info about a user

If you want to change details about a user (for a rename, or a ssh key change or password reset), you can do

ldapvi -b ou=people,dc=wikimedia,dc=org uid=$shellusername

Then you can change all the info you want to and save to persist (y to commit changes v if you want to view them first).

Tool accounts have an extra organizational unit servicegroups so to change details about a tool account you can

ldapvi -b ou=people,ou=servicegroups,dc=wikimedia,dc=org cn=tools.$toolname

and proceed the same as per user accounts.

Changing the password of a user

To change the password of a user (as root, without knowing their current password):

sudo ldappasswd -H ldap://ldap-eqiad.wikimedia.org -x -D "cn=admin,dc=wikimedia,dc=org" -W -S "uid=<USER>,ou=people,dc=wikimedia,dc=org"

You will first be asked twice for a NEW password and then once for the admin password. You can find the admin password in pwstore.

LDAP structure

As of May 2022, the main LDAP directory is structured like this:

  • dc=wikimedia,dc=org - directory root
    • ou=groups,dc=wikimedia,dc=org - groups, including Cloud VPS projects prefixed with project- and maintained by Keystone
    • ou=people,dc=wikimedia,dc=org - User accounts
    • ou=sysusers,dc=wikimedia,dc=org - A few special accounts owned by a given team, but not representing a human user, see SRE/LDAP/Sysusers for an overview
    • ou=ppolicies,dc=wikimedia,dc=org
      • cn=disabled,ou=ppolicies,dc=wikimedia,dc=org - password policy to disable an account
    • ou=profile,dc=wikimedia,dc=org - Some system accounts (including proxyagent for reads and scriptuser for writes)
    • ou=projects,dc=wikimedia,dc=org - a mess that contains both stale entries for Cloud VPS project groups and actual data (mostly sudo policies these days) that is actively being read
    • ou=servicegroups,dc=wikimedia,dc=org - Toolforge tools and some other special case "service groups"
    • ou=sudoers,dc=wikimedia,dc=org - Stale(?) OU for cloud-wide sudoers rules that is not included in the actual config used by Cloud VPS instances

See Also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=SRE/LDAP&oldid=2087028"