User:GModena (WMF)/Notes/Deploy LTR models
Models are trained on a (currently weekly) schedule by mjolnir, and uploaded to the ES indexes.
Config changes
Model names are formatted as <wiki>-YYYYMMDD-<featureset> Deploying a new model requires updating configs in wmf-config/ext-CirrusSearch.php.
- replace the old model version with the new one in wmgCirrusSearchMLRModel. This map contains multiple keys for model variants. Currently, mlr-1024rs indicates the currently deployed model, mlr-1024rs-next its successor variant. -next is relevant if we want to run A/B tests.
A/B testing
To A/B test model variants, create buckets, declare an experiment id, and enable the experiment.
- create buckets by modifying the wgCirrusSearchUserTesting map with <experiment_id> => <buckets>.. Following is an example of how a 2025-02 experiment was configured:
'wgCirrusSearchUserTesting' => [
'default' => [
// T385972
'mlr-2025' => [
'buckets' => [
'control' => [
'trigger' => 'control',
],
'mlr-2025' => [
'trigger' => 'mlr-2025',
'globals' => [
'wgCirrusSearchRescoreProfile' => 'mlr-1024rs-next',
],
],
'mlr-2025i' => [
'trigger' => 'mlr-2025i',
'globals' => [
'wgCirrusSearchInterleaveConfig' => [
'CirrusSearchRescoreProfile' => 'mlr-1024rs-next',
]
],
],
],
],
],
],
- modify, or add if it's missing, a wgCirrusSearchActiveTest that declares <wiki> => <experiment id>:
'wgCirrusSearchActiveTest' => [
'default' => null,
'dewiki' => 'mlr-2025',
'enwiki' => 'mlr-2025',
'fawiki' => 'mlr-2025',
'fiwiki' => 'mlr-2025',
'frwiki' => 'mlr-2025',
'hewiki' => 'mlr-2025',
'idwiki' => 'mlr-2025',
'itwiki' => 'mlr-2025',
'jawiki' => 'mlr-2025',
'kowiki' => 'mlr-2025',
'nlwiki' => 'mlr-2025',
'nowiki' => 'mlr-2025',
'plwiki' => 'mlr-2025',
'ptwiki' => 'mlr-2025',
'ruwiki' => 'mlr-2025',
'svwiki' => 'mlr-2025',
'viwiki' => 'mlr-2025',
'zhwiki' => 'mlr-2025',
],
Chain of patches
Break down changes into small units.
Here is an example chain of patches for the 2025-02 model deployment and experiment.
- Deploy new models: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1118782
- Create buckets for a 2025-02 experiment: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1118783
- Activate the experiment: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1118785
Deployment
Deployment happens during regular backport windows. Schedule your patches accordingly. Don't roll out everything at once.
Deploy new models and buckets first. After some monitoring, if everything is ok, enable the A/B test experiment.
During and after deployment, monitor the following:
- mwdebug logstash dashboard
- check if the model has been deploying by looking at cirrusQueryDump debug output on wiki (e.g. https://en.wikipedia.org/w/index.php?search=~test&title=Special:Search&cirrusDumpQuery). Look for an sltr key. This will tell what ranking model is currently active.
- monitor full-search latency percentiles on the -%20Full%20text%20percentiles Elasticsearch Percentiles Grafana dashboard.
Experiment deployments
- check traffic in kafka while doing a search:
- monitor the
eventlogging_SearchSatisfaction - search
~teston wiki - pass
cirrusMLRMode=<model>to the request to force executing a specific model. test that all models load to avoid log spam.
- stream from
eventlogging_SearchSatisfactionand grepevent.subTestto see if events from the test flow in - to undeploy: git revert is the preferred way