Swift/Setup New Swift Cluster (Pontoon)

This document describes how to set up a swift cluster in Pontoon, as well as a few operations that are specific to running a cluster in this virtual environment. It documents how the "swift" pontoon stack was configured, but should be fairly straightforwardly usable to make a new stack.

Set up Pontoon

The swift stack has most of the necessary configuration in it - see modules/pontoon/files/settings/swift.yaml. You'll also want to ensure that specmap.yaml has appropriate image and flavor settings (e.g. g4.cores1.ram1.disk20 for swift::proxy is sufficient and will save quota).

If you're making a new stack, you'll need to pontoonctl new-stack, pontoonctl bootstrap-stack, add the necessary git remote (which bootstrap-stack will tell you), and then pontoonctl add-rolegroup bootstrap. Once that's done and basic hosts are created, you'll need to update rolemap.yaml to lay out your cluster, which needs 3 backends and 2 frontends e.g.:

# SPDX-License-Identifier: Apache-2.0
puppetserver::pontoon:
- swift-puppet-01.swift.eqiad1.wikimedia.cloud
pki::multirootca:
- swift-pki-01.swift.eqiad1.wikimedia.cloud
puppetdb:
 - swift-puppetdb-01.swift.eqiad1.wikimedia.cloud
swift::proxy:
  - swift-ms-fe-01.swift.eqiad1.wikimedia.cloud
  - swift-ms-fe-02.swift.eqiad1.wikimedia.cloud
swift::storage:
  - swift-ms-be-01.swift.eqiad1.wikimedia.cloud
  - swift-ms-be-02.swift.eqiad1.wikimedia.cloud
  - swift-ms-be-03.swift.eqiad1.wikimedia.cloud

Create all those hosts, and run puppet enough times for it to do all the things it can successfully (which won't be everything because you don't have rings yet...).

Bootstrap Rings

Next, create initial rings, configured like prod but with min_part_hours set to 1. In a root shell in /etc/swift on the ring_manager (the lowest-numbered frontend):

swift-ring-builder account.builder create 16 3 1
swift-ring-builder container.builder create 16 3 1
swift-ring-builder object.builder create 16 3 1
swift-ring-builder object-1.builder create 16 3 1

Then, loop through the backends and add their single storage device (which is a loopback - if you can't see anything mounted at /srv/swift-storage/lda1 on the backends then run mount -a && run-puppet-agent or reboot) to each ring with weight 100, and then rebalance the rings and tar them up:

# Adjust "num" and "host" to fit your pontoon setup
for num in 1 2 3
do
  host="swift-ms-be-0${num}.swift.eqiad1.wikimedia.cloud"
  hostip=$(dig +short $host)
  zone="${num}-${hostip}"
  weight=100

  swift-ring-builder account.builder add z${zone}:6002/lda1 $weight
  swift-ring-builder container.builder add z${zone}:6001/lda1 $weight
  swift-ring-builder object.builder add z${zone}:6000/lda1 $weight
  swift-ring-builder object-1.builder add z${zone}:6000/lda1 $weight
done
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance
swift-ring-builder object-1.builder rebalance
chown swift:swift *.ring.gz

tar -czvf /var/cache/swift_rings/new_rings.tar.bz2 *.{builder,ring.gz}

Finally, copy /var/cache/swift_rings/new_rings.tar.bz2 into /srv/puppet_fileserver/volatile/swift/pontoon-swift/new_rings.tar.bz2 onto the puppet server for your stack, and run-puppet-agent everywhere. You should now have rings and a swift cluster...

Cluster Setup

If you've followed the above, you should now be able to run sudo swift-recon -r and get a sensible answer. If not, re-run puppet on the offending hosts and/or restart the swift services.

At this point, check if envoyproxy.service is running on all your frontends. If not, dpkg --purge envoyproxy && rm -rf /etc/envoy to remove the envoyproxy package entirely, and re-run puppet. There seems to be a bug whereby one of the envoy config file ends up empty on initial bootstrap and then never gets updated by puppet.

Dispersion

Set up dispersion reporting - this will take a while, but once done provides a useful check on the state of your cluster (and that e.g. reimaged nodes have been correctly backfilled):

sudo swift-dispersion-populate
sudo swift-dispersion-populate -P lowlatency

You should then be able to run sudo swift-dispersion-report and sudo swift-dispersion-report -P lowlatency from any frontend (and the cluster should report 100% dispersion).

Images for integration testing

The rewrite integration tests require a number of images (and a thumbnail, since we don't have thumbor in pontoon) to be uploaded. So download them to a suitable directory on a frontend:

Then, as root, load the mw credentials, create the necessary containers, and make the uploads:

. /etc/swift/account_AUTH_mw.env
swift post -r '.r:*' wikipedia-commons-local-thumb.02
swift post -r '.r:*' wikipedia-commons-local-public.02
swift post -r '.r:*' wikipedia-commons-local-public.3e
swift post -r '.r:*' wikipedia-commons-local-public.b3
swift upload --object-name 0/02/Berlin_2014_077.JPG/960px-Berlin_2014_077.JPG wikipedia-commons-local-thumb.02 960px-Berlin_2014_077.JPG
swift upload --object-name 0/02/Berlin_2014_077.JPG wikipedia-commons-local-public.02 Berlin_2014_077.JPG
swift upload --object-name 3/3e/Drapeau_département_fr_Corrèze.svg wikipedia-commons-local-public.3e Drapeau_département_fr_Corrèze.svg
swift upload --object-name 'b/b3/Івано-Франківськ,_Готель_Австрія,_вул._Січових_Стрільців_12.jpg' wikipedia-commons-local-public.b3 'Івано-Франківськ,_Готель_Австрія,_вул._Січових_Стрільців_12.jpg'

Testing

You can use swift stat and curl -H "Host: upload.wikimedia.org" to check these images are now downloadable, and then try the integration testing: python3 /usr/local/lib/python3.13/dist-packages/wmf/rewrite_integration_test.py (path will vary based on installed python version).

Reimaging backends

Note the IP address of a backend before destroying or reimaging it

When you destroy and recreate a backend, the "new" version of the host will have a different IP address. This means both that you need to run puppet on the other cluster nodes and sudo systemctl reload ferm.service to update firewall rules, but also that the rings need to be updated. Because this isn't an operation we do in production, the swift ring manager can't do this operation, so you need to update the rings yourself. As root on the ring manager host:

oldip=172.16.20.1 # IP of server before you destroyed it
num=2 # numeric part of server hostname
dir=$(mktemp -d)
cp /etc/swift/*.{builder,ring.gz} ${dir}/
cd $dir
weight=100
host="swift-ms-be-0${num}.swift.eqiad1.wikimedia.cloud"
hostip=$(dig +short $host)
zone="${num}-${hostip}"
swift-ring-builder account.builder add z${zone}:6002/lda1 $weight
swift-ring-builder account.builder remove --yes --ip $oldip
swift-ring-builder container.builder add z${zone}:6001/lda1 $weight
swift-ring-builder container.builder remove --yes --ip $oldip
swift-ring-builder object.builder add z${zone}:6000/lda1 $weight
swift-ring-builder object.builder remove --yes --ip $oldip
swift-ring-builder object-1.builder add z${zone}:6000/lda1 $weight
swift-ring-builder object-1.builder remove --yes --ip $oldip
swift-ring-builder account.builder rebalance
swift-ring-builder container.builder rebalance
swift-ring-builder object.builder rebalance
swift-ring-builder object-1.builder rebalance

chown swift:swift *{builder,.ring.gz}
tar -czvf /var/cache/swift_rings/new_rings.tar.bz2 *.{builder,ring.gz}

Then copy the tarball to the stack's puppet server and run puppet everywhere.