Mw-mcrouter

This is the daemonset proxying all mediawiki memcached requests, to our memcached cluster. It is running our beloved mcrouter

For how MediaWiki itself uses the cache (WANObjectCache, ParserCache, MicroStash), see Memcached for MediaWiki. For the underlying clusters, pools and servers, see Memcached.

mcrouter image and exporter

Image is in the production images repo, where the defaults are set.

Image version in production is defined in the puppet repo under profile::kubernetes::deployment_server::general:common_images

Daemonset

mw-mcrouter is running as a daemonset, i.e. every k8s node is running an instance of it.

Service

mw-mcrouter is using the mcrouter chart. Notable keys in values.yaml:

cache:mcrouter:public_service: true enables mcrouter as a standalone service
service:use_node_local_endpoints: true routes requests to the node-local endpoint of a pod
cache:mcrouter:service:clusterIPStatic IP (per DC) where the service listens, as defined in Kubernetes/Service_ips
- eqiad ClusterIP: 10.64.72.12
- codfw ClusterIP: 10.192.72.12

values-eqiad.yaml

cache:
  mcrouter:
    service:
      clusterIP: 10.64.72.12
      enabled: true
    route_prefix: eqiad/mw
    zone: eqiad
    routes:
      - route: /eqiad/mw
        pool: eqiad-servers
        failover_time: 600
      - route: /codfw/mw
        pool: codfw-servers
        failover_time: 600
      - route: /eqiad/mw-wan
        failover_time: 600
        pool: eqiad-servers
        replica:
          route: /codfw/mw-wan
          pool: codfw-servers

Deployment

In the mcrouter chart, in daemonset.yaml, this ds is configured to update a few pods at a time.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 4

During an mw-mcrouter deployment:

Generally, mcrouter's configuration or image are rarely in need for an update
In case of new image being pulled, deployment may take as long as 15'
- This is due to maxUnavailable: 1 above
Alerts for elevated mw-memcached errors
Alerts for mw-mcrouter helmfile being in a bad state
All the alerts above will clear
If deployment is stuck due to eg a node having insufficient resources to host the mcrouter

Testing changes

The safest way to test changes in mcrouter is to switch the mw-debug mediawiki deployment to use the in-pod mcrouter container. This is described in the next section

Switching Mediawiki to in-pod mcrouter container

While it sounds complicated, it is not. To switch mw-debug in eqiad to use the in-pod container, the following stanza must be added:

mw-debug/values-eqiad.yaml

cache:
  mcrouter:
    enabled: true
    route_prefix: eqiad/mw
    zone: eqiad
    routes:
      - route: /eqiad/mw
        pool: eqiad-servers
        failover_time: 600
      - route: /codfw/mw
        pool: codfw-servers
        failover_time: 600
      - route: /eqiad/mw-wan
        pool: eqiad-servers
        failover_time: 600
        replica:
          route: /codfw/mw-wan
          pool: codfw-servers

# Wikifunctions routes, omitted in production
#      - route: /local/wf
#        pool: wf-eqiad
#        # No failover for wikifunction
#        failover_time: 0
#
# use only if testing new images
# common_images:
#   mcrouter:
#     mcrouter: mcrouter:2023.07.17.00-1-20240714
#     exporter: prometheus-mcrouter-exporter:0.0.1-3-20240714

php:
  envvars:
    MCROUTER_SERVER: "127.0.0.1:11213"
    STATSD_EXPORTER_PROMETHEUS_SERVICE_HOST: false
#   MCROUTER_SERVER: "10.64.72.12:4442" # mcrouter-main.mw-mcrouter.svc.cluster.local

Replacing a host in a pool

To replace a host in a mcrouter pool (see Memcached clusters and pools and Memcached servers):

Edit profile::mediawiki::mcrouter_wancache::shards in hieradata/common/profile/mediawiki/mcrouter_wancache.yaml in the operations/puppet repo with the new server, e.g. this change.
Merge and run puppet on the active deployment server.
Redeploy mw-mcrouter via helmfile. This is fast, since it is basically a reload of mcrouter's pool configuration.
- Other changes, e.g. adding extra_args (such as in charts/mcrouter/.fixtures/mcrouter-command-line-options.yaml), would instead trigger a pod restart.

Taking a host offline for maintenance

Whether a host should be removed from the shard list, or simply left in place and allowed to fail over, depends on which pool it belongs to:

Gutter pool: if a Gutter server is going to be offline for maintenance or any other reason, it MUST be removed from the configuration. The Gutter pool is only a 3-shard-per-DC fallback (see Memcached clusters and pools), so it has no further fallback of its own if one of its own shards is unavailable.
Main pool: if a Main server is going to be offline, regardless of the expected duration, it is preferable to leave it in the shard list and let the gutter pool pick up its traffic via the normal TKO/failover mechanism (see Main route).

The reason for this is that order matters in the mcrouter shard list. Removing a server from the middle of the list causes every server listed after it to be re-hashed to a different position, i.e. all the keys previously held by those later shards effectively become cache misses and need to be recomputed and re-stored. Simply leaving a down Main host in the list and letting Gutter handle its traffic avoids this mass re-hashing/recompute, at the cost of temporarily using the Gutter pool, which is fine really.

Routes

Each MediaWiki pod accesses memcached through its local Mcrouter instance. Mcrouter introduces the concepts of routes and pools and each route applies consistent hashing on the key name to know where to send it, i.e. which of the 18 shards for memcached (see Memcached clusters and pools).

There are several routes available in our configuration, which are addressable via a route prefix that mcrouter strips from the key before forwarding the memcached command.

Main route. This route is declared as /$region/mw/ but is not addressed by MediaWiki as such. It routes to the dc-local "Main" pool shards. If a shard is perceived as unavailable from a pod ("TKO") the local mcrouter forwards all commands (incl gets, sets, and locks) to a shard of the "Gutter" pool instead.
- This route is used by the majority of traffic, through WANObjectCache::getWithSet calls in MediaWiki.
- MediaWiki doesn't use the /$region/mw/ prefix. Instead /$region/mw/ is the default route and MediaWiki sends these commands without any routing prefix.
- Switchover to and from the gutterpool is decided by Mcrouter locally. The keys stored in a gutter server have a reduced TTL.

Example

The memcached key WANCache:v:metawiki:translate-groups (belongs to the Translate extension) is formatted by the WANCache library. When Translate wants to get the value of this key, WANCache will send a GET command from MediaWiki to localhost:11213, where mcrouter is listening. The command is then further routed to mc1022 (based on key hashing). MediaWiki is totally ignorant about the mc[1,2]0XX host, it only knows about sending commands to a localhost port. A mcrouter admin command helps figure out where keys are hashed/routed to:

elukey@mw1345:~$ echo "get __mcrouter__.route(get,WANCache:v:metawiki:translate-groups)" | nc localhost 11213 -q 2
VALUE __mcrouter__.route(get,WANCache:v:metawiki:translate-groups) 0 16
10.64.0.83:11211
END

elukey@mw1345:~$ dig -x 10.64.0.83 +short
mc1022.eqiad.wmnet.

Some things to notice:

The special prefix __mcrouter__.route is intercepted by mcrouter. These are admin commands, for which proxy returns directly without contacting the memcached hosts. This function returns the target of the consistent hashing of the key name.
Mcrouter listens on port 11213 on all MediaWiki app servers, meanwhile on every mc10XX host memcached listens on port 11211.

To get a key and dump it to a file it is sufficient to:

elukey@mw1345:~$ echo "get WANCache:v:metawiki:translate-groups" | nc localhost 11213 -q 2 > dump.txt
elukey@mw1345:~$ du -hs dump.txt
380K	dump.txt

In this case the key's value is pretty big, and it needs PHP to be interpreted correctly (to unserialize it), but nonetheless we got some useful information (like the size of the key). This could be useful when it is necessary to quickly get how big a key is, rather than knowing its content.

Troubleshooting

When dealing with Kubernetes, your answer may be found in Kubernetes kubectl Cheat Sheet

Memcached server is down

That is ok, the gutter pool will pick up its traffic.

Memcached Gutter Pool server is down, and we need the Gutter Pool

In this case:

the gutter pool server in question MUST be removed from the configuration
Merge in puppet
Run puppet on the active deployment server
Deploy mw-mcrouter

Deployment is stalled

If you are deploying a new version of the daemonset, but you see pods stuck in the previous version, and elevate mw-memcached errors:

Check the daemonset's status

jiji@deploy1002:~$ kubectl get ds
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
mcrouter-main   210       210       210     210          210         <none>          30d

Check events and pods to find which node may be stalling the rollout
- If it is a resource problem (e.g. insufficient CPU), you may kill a random pod from the node in question (as root)

kube_env admin eqiad
kubectl -n mw-mcrouter get events --sort-by=.metadata.creationTimestamp
kubectl get pods --all-namespaces -o wide --field-selector spec.nodeName=wikikube-worker1001.eqiad.wmnet
kubectl -n mw-api-ext delete po mw-api-ext.eqiad.main-koko-lala

Certificates failed to renew

PKI certificates ought to use their Puppet integration to pull down renewed certificates automatically. In an emergency, it is possible to manually renew certificates using the documented process.