We run our own Docker registry at docker-registry.wikimedia.org. Internally the domain
docker-registry.discovery.wmnet is also used. The registry is used by our k8s cluster, CI, and local development.
It is highly available (
docker_registry_ha Puppet module) and backed by Swift. Although we run it active/passive because of the swift replication lag.
The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching.
Visit https://docker-registry.wikimedia.org/ to see a list of images and their tags. The listing is updated on a hourly timer and is done by the
registry-homepage-builder.py script in Puppet.
Despite the name, the docker-registry is usable by any OCI container tool, including podman. Nearly all images may be publicly downloaded, examined, run, etc. The only exception is images under the
restricted/ namespace, which contain non-disclosed security patches and require specific credentials to fetch.
Kubernetes nodes use Dragonfly to pull images.
For other docker images, like infrastructure images, we manage them using docker-pkg, see: Kubernetes/Images#Image_building
Hosts that want to upload images must be individually listed in Puppet hiera.
The upstream docker-registry software provides no access control, so it is implemented at the nginx level, which restricts GET/POST/etc. requests accordingly. As of 2021-03-18, the following accounts exist:
ci-restricted: Can pull and push any image (including "restricted/"). Used by releases servers that build the restricted MediaWiki production image.
ci-build: Can pull and push any non-restricted image. Used by contint servers via docker-pkg and the deployment pipeline.
prod-build: Can pull and push any non-restricted image. Used by build2001.codfw.wmnet via docker-pkg and build-base-images.
kubernetes: Can pull any image (including "restricted/"). Used by k8s nodes to pull images, including the restricted MediaWiki production image.
- See Kubernetes/Clusters/New#Access to restricted docker images for more details.
The passwords are all deployed using the private puppet repo. In case rotation is needed (e.g. compromise), grepping for
<name>_user_password should find all uses (switch hyphens to underscores).
docker-registry also supports authorization using JSON Web Tokens. A dedicated daemon is running which handles jwt validation. See Docker-registry/jwt-authorizer for more information.
To delete an image entirely, you may use the tool
docker-registryctl on the current build host. It will do it's best to remove the tags/image from the registry, despite the circumstances. Note: the domain used here is important.
discovery.wmnet has to be used, so you will have to adjust it if you copy and paste from the browser UI.
elukey@build2001:~$ sudo -i docker-registryctl delete-tags docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services We're about to delete the following tags for image docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services: 2021-07-28-175322-production stable Ok to proceed? (y/n)y docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:2021-07-28-175322-production[DONE] docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:stable[GONE]
There is a (not comprehensive) Httpbb test case for the docker registry:
sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2003.codfw.wmnet'