url-downloader

Please, never use this functionality to reach out to endpoints belonging to Wikimedia. You should be able to utilize the service mesh/services proxy to reach most endpoints internally in a safer, more reliable and more performing way. If you can't find out how, please reach out to SRE

url-downloader is a Squid non-caching proxy for HTTP requests. It was originally created for MediaWiki's upload-by-URL functionality, these days it is being used by any service that wants to access resources outside the boundaries of the wikimedia production realm.

There are some critical production services that depend on url-downloaders being operational, such as hCaptcha, citoid, and MediaModeration, among others. A complete list can be found in codesearch.

urldownloader low-traffic service

We have a total of four URL downloader VMs running in eqiad and codfw (urldownloader[12]00[56].wikimedia.org), which are load-balanced behind LVS.The endpoints to access them are:

urldownloader.discovery.wmnet (discovery DNS record)
urldownloader.svc.eqiad.wmnet (site-specific endpoint in eqiad)
urldownloader.svc.codfw.wmnet (site-specific endpoint in codfw)

In the past only of the URL downloader nodes was accessed via CNAME DNS records; url-downloader.$DC.wikimedia.org and url-downloader.wikimedia.org. These legacy endpoints still point to individual nodes, but should no longer be used as they will eventually be removed entirely.

For technical details please also refer to url_downloader the Puppet profile.

Troubleshooting

The service runs on port 8080 and is provided by Squid as packaged in Debian Bookworm. Cache access is logged to /var/log/squid/access.log and potential errors would end up in the journald of squid.service or /var/log/squid/cache.log. The URL downloader nodes are behind LVS, so if there's any load issues also make sure to validate if one of the nodes listed in conf-tool might be depooled (accidentally or intentionally).

To check the currently pooled servers in the cluster:

sukhe@puppetserver1001:~$ sudo confctl select 'cluster=urldownloader' get
{"urldownloader2005.wikimedia.org": {"weight": 0, "pooled": "no"}, "tags": "dc=codfw,cluster=urldownloader,service=squid"}
{"urldownloader2003.wikimedia.org": {"weight": 0, "pooled": "yes"}, "tags": "dc=codfw,cluster=urldownloader,service=squid"}
{"urldownloader2004.wikimedia.org": {"weight": 0, "pooled": "yes"}, "tags": "dc=codfw,cluster=urldownloader,service=squid"}
{"urldownloader2006.wikimedia.org": {"weight": 0, "pooled": "no"}, "tags": "dc=codfw,cluster=urldownloader,service=squid"}

The cumin alias that is used to reference the urldownloaders is A:url-downloader:

sukhe@cumin1004:~$ sudo cumin "A:url-downloader"
8 hosts will be targeted:
urldownloader[1003-1006,2003-2006].wikimedia.org
DRY-RUN mode enabled, aborting

There are is a cookbook, which can be used to perform a rolling restart/reboot: sre.misc-clusters.roll-restart-reboot-urldownloaders

Dashboards

Please see the dashboards for Host Overview and filter for urldownloader hosts, or see the Squid dashboard. There is no dashboard for the service itself yet.

urldownloader low-traffic service

Troubleshooting

Dashboards

See also