Docker-registry

We run our own Docker registry at docker-registry.wikimedia.org. Internally the domain docker-registry.discovery.wmnet is also used. The registry is used by our k8s cluster, CI, and local development.

It is highly available (docker_registry_ha Puppet module) and backed by Swift. Although we run it active/passive because of the swift replication lag.

The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching.

Browsing

Visit https://docker-registry.wikimedia.org/ to see a list of images and their tags. The listing is updated on a hourly timer and is done by the registry-homepage-builder.py script in Puppet.

Downloading images

Despite the name, the docker-registry is usable by any OCI container tool, including podman. Nearly all images may be publicly downloaded, examined, run, etc. The only exception is images under the restricted/ namespace, which contain non-disclosed security patches and require specific credentials to fetch.

Kubernetes nodes use Dragonfly to pull images.

Uploading images

For services we recommend using the Deployment pipeline which is Blubber.

For other docker images, like infrastructure images, we manage them using docker-pkg, see: Kubernetes/Images#Image_building

Hosts that want to upload images must be individually listed in Puppet hiera.

Access control

The upstream docker-registry software provides no access control, so it is implemented at the nginx level, which restricts GET/POST/etc. requests accordingly. As of 2021-03-18, the following accounts exist:

ci-restricted: Can pull and push any image (including "restricted/"). Used by releases servers that build the restricted MediaWiki production image.
ci-build: Can pull and push any non-restricted image. Used by contint servers via docker-pkg and the deployment pipeline.
prod-build: Can pull and push any non-restricted image. Used by build2001.codfw.wmnet via docker-pkg and build-base-images.
kubernetes: Can pull any image (including "restricted/"). Used by k8s nodes to pull images, including the restricted MediaWiki production image.
- See Kubernetes/Clusters/New#Access to restricted docker images for more details.

The passwords are all deployed using the private puppet repo. In case rotation is needed (e.g. compromise), grepping for <name>_user_password should find all uses (switch hyphens to underscores).

jwt-authorizer

docker-registry also supports authorization using JSON Web Tokens. A dedicated daemon is running which handles jwt validation. See Docker-registry/jwt-authorizer for more information.

Deleting images

To delete an image entirely, you may use the tool docker-registryctl on the current build host. It will do it's best to remove the tags/image from the registry, despite the circumstances. Note: the domain used here is important. discovery.wmnet has to be used, so you will have to adjust it if you copy and paste from the browser UI.

elukey@build2001:~$ sudo -i docker-registryctl delete-tags docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services
We're about to delete the following tags for image docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:
2021-07-28-175322-production
stable
Ok to proceed? (y/n)y
docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:2021-07-28-175322-production[DONE]
docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:stable[GONE]

httpbb

There is a (not comprehensive) Httpbb test case for the docker registry:

sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2003.codfw.wmnet'