Help:Using OpenStack APIs

Most Cloud VPS actions can most easily be accomplished via the Horizon web interface. If you prefer to use a commandline or are automating operations, you may want to access the OpenStack services directly.

The OpenStack APIs are extensively documented on the OpenStack doc site and elsewhere, and we will not duplicate that information here. The two cloud-vps-specific thing you absolutely need to know are:

Only authentication via OpenStack Application Credentials is supported; attempting to authenticate directly with a username and password will fail.
The auth URL for Wikimedia Cloud VPS is https://openstack.eqiad1.wikimediacloud.org:25000/v3

Application Credentials

Never store your personal developer account password in a Toolforge tool, Cloud VPS instance, or anywhere else where they might be viewed by another user. These credentials are used throughout the Wikimedia tech ecosystem, not just for Cloud VPS access. Application Credentials are safer to use as they have limited access and scope.

Even though application credentials are mostly bound to per project, they can still access some global state about your OpenStack account. If you need to store an application credential inside a shared environment, it is strongly recommended to create the credentials with a separate developer account, or at least add restrictions to limit which APIs the credentials can access.

Direct password authentication against OpenStack is prevented by the Cloud VPS auth setup. In order to access these serves outside of Horizon you will need an application credential. Application credentials provide limited-scope access to OpenStack services. A single set of credentials is limited to a particular project, and can also be restricted to a limited time period or a subset of apis or commands.

Application credentials are created in Horizon via the Identity->Application Credentials panel. You can create as many sets of credentials as you like -- it's always safer to create a new set of credentials for each use case rather than reusing one set for multiple purposes. Whenever possible, restrict your credentials to just the commands that you plan to run, and limit them to the time period in which you plan to use them.

Keep in mind that once a credential is created, you can neither edit them nor view the secret passphrase. Credentials are cheap -- if you lose access to a set, just delete and make new ones.

Interfaces

OpenStack Commandline Interfaces

OpenStack provides a commandline tool that can access most OpenStack endpoints. On Debian or Ubuntu, the package is named 'python3-openstackclient'.

The OpenStack CLI uses a plugin architecture; each commandline will look like openstack <service> <command> <args>. It is possible to include credentials in the command (e.g. --os-application-credential-secret) before the service name, but in most cases you will want to add your credentials to the shell environment or include them in a script.

Here is an example where the credentials are exported to the environment before using the CLI to list the VMs in a project.

$ apt-get install python3-openstackclient

$ export OS_AUTH_URL="https://openstack.eqiad1.wikimediacloud.org:25000/v3"
$ export OS_IDENTITY_API_VERSION=3
$ export OS_AUTH_TYPE=v3applicationcredential
$ export OS_APPLICATION_CREDENTIAL_SECRET=<redacted>
$ export OS_APPLICATION_CREDENTIAL_ID=f20108fb41324be19587c48d47bf76fe
$ export OS_APPLICATION_CREDENTIAL_NAME=clidemo
$ export OS_INTERFACE=public

$ openstack server list
+--------------------------------------+-------------------------+--------+------------------------------------------+----------------------------------------------+-----------------------+
 ID	Name	Status	Networks	Image	Flavor +--------------------------------------+-------------------------+--------+------------------------------------------+----------------------------------------------+-----------------------+
 cd0838db-578b-4fc8-b99b-be6300ef6a66	util-bullseye-codfw1dev	ACTIVE	lan-flat cloudinstances2b=172.16.128.70	debian-11.0-bullseye (deprecated 2022-06-02) g2.cores1.ram2.disk20
 b6e7ea90-7072-4536-9571-072c88c2e930	util-codfw1dev	ACTIVE	lan-flat cloudinstances2b=172.16.128.122	debian-11.0-bullseye (deprecated 2022-06-02) g2.cores1.ram2.disk20
+--------------------------------------+-------------------------+--------+------------------------------------------+----------------------------------------------+-----------------------+

The OS_AUTH_URL environment variable (or --os-auth-url commandline argument) will always point to the public Keystone endpoint. All other endpoints are discoverable via keystone, and the OpenStack cli tool will handle that discovery and query whichever other urls are needed.

OpenStack Python libraries

OpenStack also provides ready-made python bindings for accessing APIs in python code. Bindings for each project are installed separately; for example, the Nova (aka 'compute') bindings are in the Debian package python3-novaclient; the keystone (aka 'identity') bindings are in python3-keystoneclient.

There are a few unobvious things about using these bindings. First, you will always need to install the Keystone client package no matter what you're doing, because Keystone handles authentication and service discovery. Also, be aware that these packages include deprecated, service-specific commandline tools along with the python bindings; those CLIs are best ignored in favor of the much-newer integrated OpenStack client described above.

Here is some sample code of a script using keystone to authenticate, and then the nova bindings to list the VMs in a project:

andrew@buster:~/openstackclientthings$ cat appcreddemo.py 

from keystoneauth1.identity.v3 import ApplicationCredential
from keystoneauth1 import session as keystone_session
from novaclient import client as nova_client

auth = ApplicationCredential(
    auth_url="https://openstack.eqiad1.wikimediacloud.org:25000/v3",
    application_credential_secret='<redacted>',
    application_credential_id='1d083ef393ea4dafb86446da920a4c61',
    user_domain_id='default',
    )

session = keystone_session.Session(auth=auth)
client = nova_client.Client(
                '2', session=session, connect_retries=5, timeout=300,
                region_name='codfw1dev-r')
print(client.servers.list())

andrew@buster:~/openstackclientthings$ python3 ./appcreddemo.py 

[<Server: util-bullseye-codfw1dev>, <Server: util-codfw1dev>]

Gophercloud Go library

There isn't an official OpenStack SDK for Golang, but there's a community project called Gophercloud which usually works fine. There's also a library called go-cloudvps that extends Gophercloud to work with Cloud VPS specific features.

TODO: add an example for using Gophercloud

Direct API Access

Whenever possible you will want to use the ready-made bindings or CLI rather than accessing APIs directly. Nonetheless, for curl purists, or those working in non-python languages, here is a brief overview of the process:

Use the Keystone token URL (https://openstack.eqiad1.wikimediacloud.org:25000/v3/auth/tokens) to acquire a token
Parse the response for the token and the catalog. The catalog contains a list of endpoints for the different services (e.g. nova, cinder, etc.).
Include the access token in a request to the service endpoint of your choice. Routes for each endpoint are documented in the upstream OpenStack docs.

TODO: add example of using curl to get a token from an application credential

Best Practices, Realistic Expectations

Usage

Our OpenStack control plane is first and foremost designed to support interactive, human users. Many actions take seconds or minutes to perform, and there are race conditions inherent in many sequences of operations. Please keep this in mind when writing automated tooling. If you need an openstack client to act like a frenzied robot rather than a human operator clicking buttons between sips of coffee, please reach out to WMCS staff so we can coordinate and advise. We will not hesitate to block clients that monopolize or stress our infrastructure.

Uptime

Most OpenStack actions (e.g. VM creation) are infrequent and non-urgent. For this reason, the WMCS team does not prioritize API uptime as highly as things that are more public-facing like VM uptime or network access. Because API uptime is provided on a 'best effort' basis, any automated tooling or workflows that rely on the OpenStack APIs should be prepared to handle downtime gracefully and recover without human interaction; otherwise you might spend a lot of time babysitting your services during upgrade or maintenance windows (or even just middle-of-the-night outages.)

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

Chat in real time in the IRC channel #wikimedia-cloud ^connect or the bridged Telegram group
Discuss via email after you have subscribed to the cloud@ mailing list

Stay aware of critical changes and plans

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Read the News wiki page

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)