Portal:Cloud VPS/Admin/Service accounts

This page describes the current procedure/workflow/details that are required to create service accounts working on our CloudVPS (openstack) deployments.

Service accounts are user accounts intended to be used by scripts or any other kind of program or robots interacting automatically with the openstack APIs.

Setup

The setup contains several steps:

Create the account as a normal user using Bitu or Striker, i.e, a standard Developer account. Use a team- or org-wide email address (e.g. root@wmcloud.org or root+paws@wmcloud.org) rather than your personal address if available.
Decide the privileges this account should have for a given Cloud VPS project. There are mainly 2 options: member or designateadmin.
Add this new account to the Cloud VPS project it should manage, with the appropriate role.
If required, use the new user in your script!
If required, add the user to the password safelist by keystone in the keystone.conf file (for acme-chief related operations)
If required, generate openstack application credentials, and/or openstack ec2 credentials (for opentofu via gitlab CI/CD pipelines)

There are a few keys, passwords and credentials involved:

the account password, which can be used for horizon, should be stored on Pwstore on a file called cloudvps-<account-name>
- to regenerate this in eqiad1:
  - if the service account had a working email address, you may be able to use https://idm.wikimedia.org/wikimedia/password/ for password reset
  - if not, for raw manual update follow Password_and_2FA_reset#Reset_user_password_in_LDAP
- to regenerate this in codfw1dev, similar to eqiad1, but note the different LDAP server:
  - if the service account had a working email address, you may be able to use https://cloudidp-dev.wikimedia.org/wikimedia/password/ for password reset
  - if not, for raw manual update follow Password_and_2FA_reset#Reset_user_password_in_LDAP and Portal:Cloud_VPS/Admin/Testing_deployment#LDAP
for opentofu infrastructure provisioning some EC2 style credentials could be needed to store the opentofu state on Cloud VPS object store.
- to regenerate this in eqiad1: see Help:Object_storage_user_guide#S3_API, then update the gitlab CI/CD settings for pipeline variables
- to regenerate this in codf1dev: similar to eqiad1, but operating the command line from different cloudcontrol servers
for opentofu infrastrcuture provisioning, some application credentials could be needed, this is so opentofu can do RW operations on the Openstack API
- to regenerate this in eqiad1: see Help:Using_OpenStack_APIs#Application_Credentials. You need to login in https://horizon.wikimedia.org with the account password from pwstore
- to regenerate this in codfw1dev: similar to eqiad1, but horizon is at https://labtesthorizon.wikimedia.org with the codfw1dev password
the SSH key, in case the account needs it, is specified on the LDAP tree entry of each account.
- to regenerate this in eqiad1: you may be able to use the IDM website at https://idm.wikimedia.org/keymanagement/ or otherwise with manual LDAP operations, see SRE/LDAP
- to regenerate this in codfw1dev: you may be able to use the IDM website at https://cloudidp-dev.wikimedia.org or otherwise with manual LDAP operations, see Portal:Cloud_VPS/Admin/Testing_deployment#LDAP

Known examples

Infrastructure user on Cloud VPS

Sometimes you need an user to be present in all virtual machine instances on Cloud VPS.

Creating a service account means the user will be stored in LDAP, thus present in all VMs.

The password for this account may be meaningless because the user will be used for SSH (and thus, the only important thing is the SSH key).

Interacting with the openstack API

The known examples to the date are DNS-related scripts that need to interact with the desginate API.

We have examples for the acme-chief mechanism in both the deployment-prep project and in the traffic project.

$ ldap cn=deployment-prep-dns-manager \* memberOf
dn: uid=deployment-prep-dns-manager,ou=people,dc=wikimedia,dc=org
uid: deployment-prep-dns-manager
sn: Deployment-prep-dns-manager
cn: Deployment-prep-dns-manager
objectClass: inetOrgPerson
objectClass: person
objectClass: ldapPublicKey
objectClass: posixAccount
objectClass: shadowAccount
uidNumber: 19091
gidNumber: 500
homeDirectory: /home/deployment-prep-dns-manager
loginShell: /bin/false
mail: krenair+betadnsmanager@<redacted>
memberOf: cn=project-deployment-prep,ou=groups,dc=wikimedia,dc=org

In the keystone.conf file:

[auth]                                                                          
methods = external,password,token,wmtotp                                        
                                                                                
# Override the default password plugin with a custom                            
#  one that checks source IPs.                                                  
password = whitelist                                                            
                                                                                
<% @labs_networks.each do |subnet| -%>                                          
password_whitelist = novaobserver:<%=subnet%>                                   
password_whitelist = deployment-prep-dns-manager:<%=subnet%>                    
password_whitelist = traffic-cloud-dns-manager:<%=subnet%>                      
<% end -%>                                                                      
<% @prod_networks.each do |subnet| -%>                                          
password_whitelist = *:<%=subnet%>                                              
<% end -%>

Role assignment:

root@cloudcontrol1004:~# openstack role add --user deployment-prep-dns-manager --project deployement-prep designateadmin
root@cloudcontrol1004:~# openstack role add --user deployment-prep-dns-manager --project deployement-prep observer
root@cloudcontrol1003:~# openstack role assignment list --names --user deployment-prep-dns-manager
+----------------+-------------------------------------+-------+-------------------------+--------+-----------+
| Role           | User                                | Group | Project                 | Domain | Inherited |
+----------------+-------------------------------------+-------+-------------------------+--------+-----------+
| observer       | Deployment-prep-dns-manager@Default |       | deployment-prep@Default |        | False     |
| designateadmin | Deployment-prep-dns-manager@Default |       | deployment-prep@Default |        | False     |
+----------------+-------------------------------------+-------+-------------------------+--------+-----------+
root@cloudcontrol1003:~# openstack role assignment list --names --user traffic-cloud-dns-manager
+----------------+-----------------------------------+-------+-----------------+--------+-----------+
| Role           | User                              | Group | Project         | Domain | Inherited |
+----------------+-----------------------------------+-------+-----------------+--------+-----------+
| observer       | Traffic-cloud-dns-manager@Default |       | traffic@Default |        | False     |
| designateadmin | Traffic-cloud-dns-manager@Default |       | traffic@Default |        | False     |
+----------------+-----------------------------------+-------+-----------------+--------+-----------+

Opentofu operations from Gitlab CI/CD

If you want to operate Cloud VPS infrastructure using a service account from Gitlab CI/CD, you may want to:

create a service account, as described above
associate the service account to the target project via a role assignment, as described above
generate ec2 credentials, if you want to store the tofu state in Cloud VPS object store
generate application credencials, for RW access to the openstack API. This needs to be done from Horizon, logged in as the service account
- you may want to generate a password using the 'Developer Account LDAP password reset' procedure

Examples:

user@cloudcontrol1006:~ $ sudo wmcs-openstack --os-project-id testlabs ec2 credential create --user srv-networktests
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                                                          |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| access     | 0bbb318cdcea4f88b3a41b6384326XXX                                                                                                               |
| links      | {'self': 'https://openstack.eqiad1.wikimediacloud.org:25357/v3/users/srv-networktests/credentials/OS-EC2/0bbb318cdcea4f88b3a41b6384326XXX'} |
| project_id | testlabs                                                                                                                                       |
| secret     | YYYZZZHHHBBB123654AAAGGGJJJ00XXX                                                                                                               |
| trust_id   | None                                                                                                                                           |
| user_id    | srv-networktests                                                                                                                               |
+------------+------------------------------------------------------------------------------------------------------------------------------------------------+

For the credentials to be valid, the user must have a role assigned in the project:

user@cloudcontrol1006:~ $ sudo wmcs-openstack role add --project testlabs --user srv-networktests member

Troubleshooting

If your script gets an error when using the API like this:

keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-c6f30f48-c91b-4681-a263-40df8df101df)

Or in the server, you see logs like these:

(keystone.middleware.auth): 2019-08-07 08:57:55,636 WARNING RBAC: Invalid token
(keystone.common.wsgi): 2019-08-07 08:57:55,639 WARNING The request you have made requires authentication.
(wmfkeystoneauth.password_whitelist): 2019-08-07 09:00:02,466 WARNING Password auth not allowed for traffic-cloud-dns-manager from 172.16.7.184
(keystone.common.wsgi): 2019-08-07 09:00:02,467 WARNING Authorization failed. The request you hav

This means that you need to add the user to the password_safelist setting in keystone.conf.

Password reset

See setup described above, which should contain info on how to refresh different credentials.