We run our own Docker registry at docker-registry.wikimedia.org. Internally the domain
docker-registry.discovery.wmnet is also used. The registry is used by our k8s cluster, CI, and local development.
It is highly available (
docker_registry_ha Puppet module) and backed by Swift. Although we run it active/passive because of the swift replication lag.
The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching.
Visit https://docker-registry.wikimedia.org/ to see a list of images and their tags. The listing is updated on a hourly timer and is done by the
registry-homepage-builder.py script in Puppet.
Despite the name, the docker-registry is usable by any OCI container tool, including podman. Nearly all images may be publicly downloaded, examined, run, etc. The only exception is images under the
restricted/ namespace, which contain non-disclosed security patches and require specific credentials to fetch.
Kubernetes nodes use Dragonfly to pull images.
For services we recommend using the Deployment pipeline which is Blubber.
For other docker images, like infrastructure images, we manage them using docker-pkg, see: Kubernetes/Images#Image_building
Hosts that want to upload images must be individually listed in Puppet hiera.
The upstream docker-registry software provides no access control, so it is implemented at the nginx level, which restricts GET/POST/etc. requests accordingly. As of 2021-03-18, the following accounts exist:
ci-restricted: Can pull and push any image (including "restricted/"). Used by releases servers that build the restricted MediaWiki production image.
ci-build: Can pull and push any non-restricted image. Used by contint servers via docker-pkg and the deployment pipeline.
prod-build: Can pull and push any non-restricted image. Used by build2001.codfw.wmnet via docker-pkg and build-base-images.
kubernetes: Can pull any image (including "restricted/"). Used by k8s nodes to pull images, including the restricted MediaWiki production image.
- See Kubernetes/Clusters/New#Access to restricted docker images for more details.
The passwords are all deployed using the private puppet repo. In case rotation is needed (e.g. compromise), grepping for
<name>_user_password should find all uses (switch hyphens to underscores).
docker-registry also supports authorization using JSON Web Tokens. A dedicated daemon is running which handles jwt validation. See Docker-registry/jwt-authorizer for more information.
To delete an image entirely, you may use the tool
docker-registryctl on the current build host. It will do it's best to remove the tags/image from the registry, despite the circumstances.
elukey@build2001:~$ sudo -i docker-registryctl delete-tags docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services We're about to delete the following tags for image docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services: 2021-07-28-175322-production stable Ok to proceed? (y/n)y docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:2021-07-28-175322-production[DONE] docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:stable[GONE]
There is a (not comprehensive) Httpbb test case for the docker registry:
sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2003.codfw.wmnet'