GitLab/Object/Storage

GitLab stores some data in a Ceph-based S3-compatible object storage called APUS. This reduces disk usage and shortens backup and restore cycles for the GitLab hosts.

The production instance uses credentials with read-write access and the replicas with read-only access.

Artifacts

This files are generated in CI jobs and contain job logs, build artifacts and other files. This files are stored in a bucket called gitlab-artifacts on the APUS Ceph cluster. All GitLab instances (production and replicas) share the same bucket and data. Replicas use read-only credentials. This also means during a failover no artifacts have to be synchronized.

See also https://docs.gitlab.com/ci/jobs/job_artifacts/.

Important docs Migrating to object storage.

Packages

This files are also generated in CI jobs but contain bigger files like compiled applications, releases, Debian packages. These files are stored in a bucket called gitlab-packages on the APUS Ceph cluster, see T378922.

See also https://docs.gitlab.com/administration/packages/.

Important docs Migrate packages between object storage and local storage.

Accessing the Ceph buckets

Direct access to the buckets is possible using tools like s3cmd. This tool is installed on newer cumin hosts and can be used when a .s3cfg file is populated with the correct credentials (which can be found in private puppet hieradata/common/profile/ceph/s3/client.yaml).

s3cmd can be used to list or delete files, create buckets or change acls:

jelto@cumin2002:~$ s3cmd mb gitlab-artifacts
jelto@cumin2002:~$ s3cmd mb gitlab-packages

jelto@cumin2002:~$ s3cmd ls
2025-05-27 08:32  s3://gitlab-artifacts
2025-05-27 08:32  s3://gitlab-packages

jelto@cumin2002:~$ s3cmd ls s3://gitlab-artifacts
...

See also https://phabricator.wikimedia.org/T378922, this is work in progress

Bucket backups

In general the data stored on object storage are expendable and could be re-created. To reduce the work of recreating all packages they are also in the backup rotation.

As a workaround until proper bucket backups are available the s3://gitlab-packages bucket is cloned to /srv/gitlab-backups/packages-mirror daily using the ceph::client::sync_local module. The modules uses s3cmd to sync the files to a local folder. This local folder is in the bacula config and backed-up with the default backup rotation ("monthly" with daily incrementals).