url-downloader

Please, never use this functionality to reach out to endpoints belonging to Wikimedia. You should be able to utilize the service mesh/services proxy to reach most endpoints internally in a safer, more reliable and more performing way. If you can't find out how, please reach out to SRE

url-downloader is a Squid non-caching proxy for HTTP requests. It was originally created for MediaWiki's upload-by-URL functionality, these days it is being used by any service that wants to access resources outside the boundaries of the wikimedia production realm. It is not a highly available service and as such no assumptions should be made in code regarding availability.

Service

See also url_downloader in Puppet.

For eqiad: Hosted on urldownloader100[34].wikimedia.org
For codfw: Hosted on urldownloader200[34].wikimedia.org

In both datacenters only one url-downloader is active at any time, and is pointed at via the DNS record for url-downloader.$DC.wikimedia.org, and exposed at url-downloader.$DC.wikimedia.org:8080.

url-downloader.wikimedia.org:8080 points to eqiad for backward-compatibility.

In MediaWiki configuration, it would be typically obtained via $wmgLocalServices['urldownloader'] (which also provides an appropriate proxy URL on beta).

Administration

Adding/replacing URL downloader nodes

Any server in the production networks can directly access a new URL downloader proxy after applying the Puppet role. No changes in homer/on the routers are necessary. Access to a specific node can be tested with e.g.

curl -x http://urldownloader100X.wikimedia.org:8080 $URL > download

In addition the new node (or the node to be removed) need to be added/removed from the egress rules for Kubernetes. An example commit can be found at https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1303341. Following the merge this need to be deployed via the admin_ng section of the deployment charts. This enables access to all our Kubernetes clusters, not just wikikube.

sudo -i
cd /srv/deploymenmt/deployment-charts/helmfile.d/admin_ng
helmfile -e [staging-eqiad|staging-codfw|eqiad|codfw] -i [diff|apply]

The change need to be applied to all external-services and calico.