Docker-registry
We run our own Docker registry at docker-registry.wikimedia.org. Internally the domain docker-registry.discovery.wmnet is also used. The registry is used by our k8s cluster, CI, and local development.
It is highly available (docker_registry_ha Puppet module) and backed by Swift. Although we run it active/passive because of the swift replication lag.
The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching.
Browsing
Visit https://docker-registry.wikimedia.org/ to see a list of images and their tags. The listing is updated on a hourly timer and is done by the registry-homepage-builder.py script in Puppet.
Downloading images
Despite the name, the docker-registry is usable by any OCI container tool, including podman. Nearly all images may be publicly downloaded, examined, run, etc. The only exception is images under the restricted/ namespace, which contain non-disclosed security patches and require specific credentials to fetch.
Kubernetes nodes use Dragonfly to pull images.
Uploading images
For services we recommend using the Deployment pipeline which is Blubber.
For other docker images, like infrastructure images, we manage them using docker-pkg, see: Kubernetes/Images#Image_building
Hosts that want to upload images must be individually listed in Puppet hiera.
Access control
The upstream docker-registry software provides no access control, so it is implemented at the nginx level, which restricts GET/POST/etc. requests accordingly. As of 2021-03-18, the following accounts exist:
ci-restricted: Can pull and push any image (including "restricted/"). Used by releases servers that build the restricted MediaWiki production image.ci-build: Can pull and push any non-restricted image. Used by contint servers via docker-pkg and the deployment pipeline.prod-build: Can pull and push any non-restricted image. Used by build2001.codfw.wmnet via docker-pkg and build-base-images.kubernetes: Can pull any image (including "restricted/"). Used by k8s nodes to pull images, including the restricted MediaWiki production image.- See Kubernetes/Clusters/New#Access to restricted docker images for more details.
The passwords are all deployed using the private puppet repo. In case rotation is needed (e.g. compromise), grepping for <name>_user_password should find all uses (switch hyphens to underscores).
jwt-authorizer
docker-registry also supports authorization using JSON Web Tokens. A dedicated daemon is running which handles jwt validation. See Docker-registry/jwt-authorizer for more information.
Deleting images
To delete an image entirely, you may use the tool docker-registryctl on the current build host. It will do it's best to remove the tags/image from the registry, despite the circumstances. Note: the domain used here is important. discovery.wmnet has to be used, so you will have to adjust it if you copy and paste from the browser UI.
elukey@build2001:~$ sudo -i docker-registryctl delete-tags docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services
We're about to delete the following tags for image docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:
2021-07-28-175322-production
stable
Ok to proceed? (y/n)y
docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:2021-07-28-175322-production[DONE]
docker-registry.discovery.wmnet/wikimedia/machinelearning-liftwing-inference-services:stable[GONE]
Maintenance
The docker-registry nodes consist of the docker registry itself as well as an nginx reverse-proxy in front to handle authentication as well as local caching.
Service management
The following units run on the registry nodes:
| Service unit | Description |
|---|---|
docker-registry-ha-jwt.service |
JSON Web Token authoriser |
docker-registry-swift.service |
Registry instance backed by Swift |
docker-registry-ml.service |
Registry instance serving the ML/LiftWing namespace |
docker-registry-restricted.service |
Registry instance serving restricted images |
Restarting
Before restarting any service, confirm which node is currently active and prefer restarting the passive node where possible to avoid unnecessary disruption.
cumin1003:$ sudo confctl --object-type discovery select 'dnsdisc=docker-registry' get
sudo systemctl restart docker-registry-ha-jwt.service
sudo systemctl restart docker-registry-swift.service
sudo systemctl restart docker-registry-ml.service
sudo systemctl restart docker-registry-restricted.service
To check status:
sudo systemctl status docker-registry-ha-jwt.service
Note that nginx sits in front of all registry instances and handles authentication and local caching.
sudo systemctl status nginx.service
sudo systemctl restart nginx.service
Post-restart verification - httpbb
Run the httpbb tests against the restarted node to confirm it is serving correctly. Make sure the instance is not in read-only mode first (profile::docker_registry::read_only_mode: false):
sudo httpbb /srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml --hosts 'registry2004.codfw.wmnet'