Jump to content

Help:Object storage user guide

From Wikitech

This page explains how to use the Swift and S3-compatible object storage service in Cloud VPS. This service is provided by the Ceph rados gateway.

What is object storage?

Object storage treats entire files as the largest addressable unit. Files can be created, read, or deleted via REST apis but cannot be edited or randomly accessed. When an object is created in a public container, it can be accessed on the open internet via a single URL.

Object storage is most often used to store static content for web sites, but has many other uses.

Considerably more context on the subject is available on English Wikipedia.

Interacting with the Cloud VPS object storage service

There are two main interfaces for interacting with the object storage service:

  • The OpenStack Swift-like API
  • The Amazon S3-like API

Horizon and other OpenStack-native clients will use the Swift interface, but support for the S3 interface is much more common than support for the Swift interface in third-party software. In general, you should not mix the two interfaces on the same bucket, as access control lists and similar don't directly translate between each other.

Horizon

Logged-in project members can access Object Storage via the 'Object Store' tab in Horizon.

Horizon web interface displaying the Object store tab

Each object is stored in a container. Containers can be marked as public or private. Files in public containers will be viewable to the entire internet; access to private containers is limited to users with authorized API access.

Horizon web interface showing container details of the Object store tab

Horizon will display a link to the URL for public containers. All files are visible as subpaths of that URL.

OpenStack CLI

Objects and containers can be accessed via the OpenStack commandline. The process of authorization and access is similar to that for other openstack services and access is only available via application credentials. Containers are accessed via 'openstack container' subcommands, and objects via 'openstack object' subcommands.

$ cat ./smallfile.txt
I am a small file

$ openstack --os-cloud cloudvps container create mycontainer --public
+---------------+-------------+-----------------------------------------------------+
| account       | container   | x-trans-id                                          |
+---------------+-------------+-----------------------------------------------------+
| AUTH_testlabs | mycontainer | tx00000ebe291249fc76624-00652837dc-2b59b075-default |
+---------------+-------------+-----------------------------------------------------+

$ openstack --os-cloud cloudvps container show mycontainer
+----------------+-------------------+
| Field          | Value             |
+----------------+-------------------+
| account        | AUTH_testlabs     |
| bytes_used     | 18                |
| container      | mycontainer       |
| object_count   | 1                 |
| read_acl       | .r:*,.rlistings   |
| storage_policy | default-placement |
+----------------+-------------------+

$ openstack --os-cloud cloudvps object create mycontainer ./smallfile.txt
+-----------------+-------------+----------------------------------+
| object          | container   | etag                             |
+-----------------+-------------+----------------------------------+
| ./smallfile.txt | mycontainer | ee02f0e585723ad3ea8f3f3bedf711c9 |
+-----------------+-------------+----------------------------------+

$ openstack --os-cloud cloudvps object show mycontainer smallfile.txt
+----------------+----------------------------------+
| Field          | Value                            |
+----------------+----------------------------------+
| account        | AUTH_testlabs                    |
| container      | mycontainer                      |
| content-length | 18                               |
| content-type   | text/plain                       |
| etag           | ee02f0e585723ad3ea8f3f3bedf711c9 |
| last-modified  | Thu, 12 Oct 2023 18:16:53 GMT    |
| object         | smallfile.txt                    |
+----------------+----------------------------------+

$ curl https://object.eqiad1.wikimediacloud.org/swift/v1/AUTH_testlabs/mycontainer/smallfile.txt
I am a small file

Swift API

The OpenStack Swift API endpoint mostly behaves like any other OpenStack API on Cloud VPS. See Help:Using OpenStack APIs for details. As with the other APIs, you should use a dedicated developer account for any credentials that are stored in shared environments (like in Cloud VPS or Toolforge).

S3 API

The API also supports an Amazon S3 compatible API, which seems to be the industry standard for various software to use. The Ceph documentation contains details about particular quirks of the API. The base S3 API URL is https://object.eqiad1.wikimediacloud.org, and the S3 region is default. The public URL for each bucket (if configured with a public ACL) is https://object.eqiad1.wikimediacloud.org/PROJECT:BUCKET/.

The Cloud VPS setup is configured to use Keystone-backed authentication which means that you interact with the OpenStack API to create AWS-style authentication tokens. Again, you should use a dedicated developer account for any credentials that are stored in shared environments (like in Cloud VPS or Toolforge).

To create an AWS/ec2 style credential, you need an authenticated OpenStack CLI. This token will be bound to the current user and project.

$ openstack ec2 credential create
+------------+-----------------------------------------------------------------------------------------------------------+
| Field      | Value                                                                                                     |
+------------+-----------------------------------------------------------------------------------------------------------+
| access     | someid                                                                                                    |
| links      | {'self': 'https://openstack.eqiad1.wikimediacloud.org:25000/v3/users/someuser/credentials/OS-EC2/someid'} |
| project_id | someproject                                                                                               |
| secret     | somesecret                                                                                                |
| trust_id   | None                                                                                                      |
| user_id    | someuser                                                                                                  |
+------------+-----------------------------------------------------------------------------------------------------------+

Take note of the access and secret values.

Quotas and other limitations

By default, any project may store up to 4096 different files and use a total of 8GB of space. These files may be distributed over any number of containers.

If you require more storage space, open a quota request ticket.

Toolforge

Right now there's no specific way for Toolforge users to use the object storage service. If you have a use case in Toolforge that would benefit from object storage, requesting a Cloud VPS project for that purpose only is possible.

Data persistence

Cloud VPS Object storage is not backed up, and is stored using 'erasure coding,' a slightly less redundant version of Ceph storage than that used for VMs or Cinder. The object store should probably not be the only place you keep your critical data.

What doesn't work

  • Object storage will not work at all for projects with a - in their name due to limitations in the software. Feel free to open a project request for a new project with a simpler name, either to replace your project or to use alongside your current project for object storage.
  • Large files (greater than about 200M) cannot be uploaded from the Horizon UI. Large file support works fine via the openstack CLI using application credentials.

See also

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)