Jump to content

Portal:Cloud VPS/Admin/Projects lifecycle

From Wikitech

This page describes how Cloud VPS projects are supposed to be created and deleted, i.e, the lifecycle. In the OpenStack world, project is known as tenant.

Working with projects

Rather than adding yourself as a project admin for a project to visualize / work on it, add yourself as a domain admin: 

$ sudo wmcs-openstack role add --user USERNAME --domain default --inherited admin

To see which projects you have a membership in see, https://openstack-browser.toolforge.org/user/USERNAME

Requests

Requests for new projects or for increased quotas on existing projects are made by users in Phabricator. These requests are reviewed and fulfilled by the current member on clinic duty.

Before fulfilling the request, please ask a different member of the team to leave a "+1" comment in the Phabricator task. In this way we make sure that at least two people in the team review each request.

Creating a new project

You can use the cookbook for it, and it will take care of doing the checks and guiding you through the process, you can pass users to add, and quotas to set, see the help for all the options: 

root@cloudcumin1001:~$ cookbook wmcs.vps.create_project --help


See the following collapsed section for the full manual steps:

Manual steps


Prerequisites:

  • Project names should be all lower case, if not some errors will happen when requesting the puppet certificate
  • Project names should not contain underscores or any other characters that are not valid in DNS names.
  • Users must have valid Developer accounts
  • You have added the project to the list in Portal:Cloud VPS/Admin/Projects with the relevant contacts, tags, and link to the creation task.

Steps:

  1. Make sure the project name does not clash with an existing DNS record (phab:T360294): 
    cloudcontrol1005:~$ sudo wmcs-openstack recordset list --all-projects wmcloud.org. |grep {NEW_PROJECT_NAME}.wmcloud.org
  2. Make sure the users requesting the project exists in LDAP 
    cloudcontrol1005:~$ sudo wmcs-openstack user list | grep -i {USERNAME}
  3. Create the project, via the CloudVPS tofu project, see https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/OpenTofu
  4. Modify initial quotas if required (see section below)
  5. Add initial users to roles: 
    $ sudo cookbook wmcs.vps.add_user_to_project --project newp --user requestinguser --as-member --task-id TXXXXXX
  6. Please make sure to instruct the user to join the cloud-announce mailing list.

Deleting a project

Project deletion tends to leave orphaned resources lying about. Eventually this should all be handled by Designate or Keystone hooks, but until then:

  1. Make sure there are no instances in the project. This can be done in Horizon or via the commandline:
    $ sudo wmcs-openstack server list --project <project>
$ sudo wmcs-openstack server delete <instance id>
  2. Note if a custom flavor is present (these should be standardized after September 2020), and remove if no longer used outside of project:
    $ sudo wmcs-openstack flavor list
$ sudo wmcs-openstack flavor delete <flavor name>
  3. Make sure the project is not configured for NFS: modules/cloudnfs/data/projects.yaml.
  4. Make sure there are no dns zones allocated to the project. This can be done in Horizon Here (Be sure to select the right project using the dropdown) or via the commandline:
    $ sudo wmcs-openstack zone list --sudo-project-id <project>
$ sudo wmcs-openstack zone delete --sudo-project-id <project> <zone id>
  5. Delete any proxies the project may have via Horizon. Be sure to select the right project using the dropdown
  6. Delete any prefix- or project-wide puppet configurations via Horizon Here and Here. Be sure to select the right project using the dropdown
  7. Utilizing horizon, confirm no project specific config, rules, or resources remain.
  8. Finally, delete the project using OpenTofu, see Portal:Cloud_VPS/Admin/OpenTofu
  9. Remove the project from the list at Portal:Cloud VPS/Admin/Projects

Manage project access

Note that project access should usually be self-managed by project admins on Horizon. Be careful to avoid phishing attacks if you're adjusting them via the CLI tools! The following commands are recorded here for reference.

Add access

There are essentially two roles that are interesting most of the times: reader (lets the user SSH in to any instances) and member (lets the user manage project resources via Horizon). For more information, see: Help:Cloud services user roles and rights.

Using the preferred cookbook (defaults to reader role, add --as-member for member role): 

cloudcumin1001:~$ sudo cookbook wmcs.vps.add_user_to_project \
    --task-id sometask \
    --project someproject \
    --user someuser

Or, as a fallback, manually: 

cloudcontrol1005:~$ sudo wmcs-openstack role add --project someproject --user someuser member

Listing users and roles for a project: 

cloudcontrol1005:~$ sudo wmcs-openstack role assignment list --names --project someproject

Remove access

If you need to drop some user involvement with a project, using the preferred cookbook: 

$ cookbook wmcs.vps.remove_user_from_project \
    --cluster-name eqiad1 \
    --task-id sometask \
    --project someproject \
    --user someuser

Or, as a fallback, manually: 

$ sudo wmcs-openstack role assignment list --names --user someuser
:# OR
$ sudo wmcs-openstack role assignment list --names --project someproject
:# and then accordingly for all the roles
$ sudo wmcs-openstack role remove --user someuser --project someproject somerole

Modifying project quotas

CURRENTLY RECOMMENDED: You can use the following cookbook to increase a project's quotas. Be mindful that the values you specify in the cookbook are the amount of increase, not the final values. 

$ ssh cloudcumin1001.eqiad.wmnet
cloudcumin1001:~$ sudo cookbook wmcs.openstack.quota_increase -h
usage: cookbook [GLOBAL_ARGS] wmcs.openstack.quota_increase [-h] [--project PROJECT] [--task-id TASK_ID] [--no-dologmsg] --cluster-name {eqiad1,codfw1dev}
[--backup-gigabytes BACKUP_GIGABYTES] [--backups BACKUPS] [--cores CORES] [--fixed-ips FIXED_IPS]
[--floating-ips FLOATING_IPS] [--gigabytes GIGABYTES] [--gigabytes_standard GIGABYTES_STANDARD]
[--groups GROUPS] [--injected-file-size INJECTED_FILE_SIZE] [--injected-files INJECTED_FILES]
[--injected-path-size INJECTED_PATH_SIZE] [--instances INSTANCES] [--key-pairs KEY_PAIRS]
[--networks NETWORKS] [--per-volume-gigabytes PER_VOLUME_GIGABYTES] [--ports PORTS]
[--properties PROPERTIES] [--ram RAM] [--rbac_policies RBAC_POLICIES] [--routers ROUTERS]
[--secgroup-rules SECGROUP_RULES] [--secgroups SECGROUPS] [--server-group-members SERVER_GROUP_MEMBERS]
[--server-groups SERVER_GROUPS] [--snapshots SNAPSHOTS] [--snapshots_standard SNAPSHOTS_STANDARD]
[--subnet_pools SUBNET_POOLS] [--subnets SUBNETS] [--volumes VOLUMES]
[--volumes_standard VOLUMES_STANDARD]

WMCS openstack - increase a project's quota by a given amount

If talking about memory, things like 10G/250M are supported.

Usage example: wmcs.openstack.quota_increase \
    --project admin-monitoring \
    --gigabytes 30G \
    --cluster-name eqiad1 \
    --instances 5

Tracking project quotas on tofu-infra repository

The tofu-infra repository can override quota. The quotas you set via tofu-infra will override those set via the openstack CLI or the wmcs.openstack.quota_increase cookbook, mostly because tofu will reconcile them on every run, thus reverting any changes made by any other tooling.

That being said, tofu-infra only track overrides that have been explicitly created for a given project in the repository. This means that if no quotas for a given project have been added to tofu-infra, you are free to update the quotas via other methods (cookbook, CLI) and tofu wont revert them.

However, please note some quotas may not be supported via tofu-infra, for a variety of reasons, including:

  • logic not added to the tofu-infra repository
  • no support in the opentofu openstack provider, so logic cannot be added to the repository

See also:

Modifying the quotas with the openstack CLI

If the cookbook above is not working, or if you need to decrease quotas, you can set quotas using the openstack CLI from a cloudcontrol host. Be mindful the values here are the final values, not the amount of increase.

//check current values
cloudcontrol1005:~ $ sudo wmcs-openstack quota show videocuttool
//set desired amounts
cloudcontrol1005:~ $ sudo wmcs-openstack quota set --cores 26 --ram 36864 --gigabytes 320 --instances 8 videocuttool

Database / Trove quotas

See Trove_quotas

swift / S3 / radosgw / object storage

For setting object storage quotas, use the radosgw-admin tool.

Check the user list. Note that radosgw users are created on demand, so if you don't see the user here you can nudge things with (for example) OS_PROJECT_ID=<project> openstack container create newcontainer: 

user@cloudcontrol1005:~ $ sudo radosgw-admin user list
[
    "cloudinfra$cloudinfra",
    "integration$integration",
    "mwstake$mwstake",
    "deployment-prep$deployment-prep",
    "automation-framework$automation-framework",
    "wikiapiary$wikiapiary",
    "language$language",
    "wikisp$wikisp",
    "foundationmemory$foundationmemory",
    "[..]",
    "analytics$analytics"
]

Check the quota for a given user: 

user@cloudcontrol1005:~ $ sudo radosgw-admin user info --uid qrank\$qrank
{
    "user_id": "qrank$qrank",
    "display_name": "qrank",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": true,
        "check_on_raw": false,
        "max_size": 8589934592,
        "max_size_kb": 8388608,
        "max_objects": 4096
    },
    "temp_url_keys": [],
    "type": "keystone",
    "mfa_ids": []
}

Check user stats (used quota): 

user@cloudcontrol1005:~ $ sudo radosgw-admin user stats --uid qrank\$qrank
{
    "stats": {
        "size": 8622484329,
        "size_actual": 8622555136,
        "size_kb": 8420395,
        "size_kb_actual": 8420464,
        "num_objects": 39
    },
    "last_stats_sync": "2024-03-14T15:35:48.996432Z",
    "last_stats_update": "2024-03-15T07:25:32.330966Z"
}

Finally, modify quotas: 

user@cloudcontrol1005:~ $ sudo radosgw-admin quota set --quota-scope=user --uid=qrank\$qrank --max-size=20G

See also upstream docs: https://docs.ceph.com/en/latest/radosgw/admin/#quota-management

History

In previous versions of Cloud VPS, the project lifecycle was handled via Wikitech itself, with some custom MediaWiki plugins to hook into the Openstack API. This was eventually replaced by Horizon, which can handle all the operations by itself.

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Portal:Cloud_VPS/Admin/Projects_lifecycle&oldid=2292996"