We run our own Docker registry at docker-registry.wikimedia.org. Internally the domain
docker-registry.discovery.wmnet is also used. It is highly available (
docker_registry_ha Puppet module) and backed by Swift. The registry is used by our k8s cluster, CI, and local development.
The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching.
Visit https://docker-registry.wikimedia.org/ to see a list of images and their tags. The listing is updated on a hourly timer and is done by the
registry-homepage-builder.py script in Puppet.
Despite the name, the docker-registry is usable by any OCI container tool, including podman. Nearly all images may be publicly downloaded, examined, run, etc. The only exception is images under the
restricted/ namespace, which contain non-disclosed security patches and require specific credentials to fetch.
Kubernetes nodes use Dragonfly to pull images.
For other docker images, like infrastructure images, we manage them using docker-pkg, see: Kubernetes/Images#Image_building
Hosts that want to upload images must be individually listed in Puppet hiera.
The upstream docker-registry software provides no access control, so it is implemented at the nginx level, which restricts GET/POST/etc. requests accordingly. As of 2021-03-18, the following accounts exist:
ci-restricted: Can pull and push any image (including "restricted/"). Used by releases servers that build the restricted MediaWiki production image.
ci-build: Can pull and push any non-restricted image. Used by contint servers via docker-pkg and the deployment pipeline.
prod-build: Can pull and push any non-restricted image. Used by deneb.codfw.wmnet via docker-pkg and build-base-images.
kubernetes: Can pull any image (including "restricted/"). Used by k8s nodes to pull images, including the restricted MediaWiki production image.
- See Kubernetes/Clusters/New#Access to restricted docker images for more details.
The passwords are all deployed using the private puppet repo. In case rotation is needed (e.g. compromise), grepping for
<name>_user_password should find all uses (switch hyphens to underscores).
To delete an image entirely, you may use the tool
docker-registryctl on the current build host. It will do it's best to remove the tags/image from the registry, despite the circumstances.
elukey@deneb:~$ docker-registryctl delete-tags docker-registry.discovery.wmnet/istio/operator:1.6* docker-registry.discovery.wmnet/istio/operator:1.6.14-1 [DONE]
There is a (not comprehensive) Httpbb test case for the docker registry:
sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2008.codfw.wmnet'