User:Razzi/A week with the search team

From Wikitech

Elasticsearch: add a ?v to view headings

razzi@datahubsearch1001:~$ curl localhost:9200/_cat/health
1647383886 22:38:06 datahub green 3 3 true 0 0 0 0 0 0 - 100.0%
razzi@datahubsearch1001:~$ curl localhost:9200/_cat/health?v
epoch      timestamp cluster status node.total node.data discovered_master shards pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1647383888 22:38:08  datahub green           3         3              true      0   0    0    0        0             0                  -                100.0%

Elasticsearch shard config

Usually there are 3 shards, and the number of shards is configured in mediawiki-config in the hilariously monolithic https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php#L24656

Reached out to Riccardo to deploy and test spicerack change

Going in, my goal was to learn how to deploy spicerack myself, but according to him, it is by design that only Infrastructure Foundations deploys spicerack. Not what I'd recommend but they've been timely enough so far :)

Cumin1001 was reimaged, deleting my home directory

Good thing I backed up my spicerack_repl.py.

LVS services use direct routing (LVS-DR) and are defined in `hieradata/common/service.yaml`.

Services follow a transition path of

  • service_setup
  • lvs_setup
  • monitoring_setup
  • production

vagrant is the integration environment for cirrussearch

but it's complicated and falling out of favor; as people are using docker since mediawiki-vagrant is complicated.

Spicerack: use tox to run local tests

tox -e py39-unit for unit tests and tox -e py3-flake8

tox -e py39-unit for unit tests and tox -e py3-flake8 (or prospector, for the various linting tools. You can also run more than one at a time with tox -e py39-flake8,py39-prospector,py39-bandit

I spread the joy of single-meeting urls

Now they are going to use the same meeting url for their standard meetings. They just need to come up with a cool name for it (sorry, "The Batcave" and "The Tardis" are already taken by data engineering) :)

Search team projects board

https://easyretro.io/publicboard/jPibkLBsemdD7MdOXxf9gI9Lf1D3/ad3c74ae-d2be-427d-8474-15084b252b51

Just for fun

Mjolnir is Thor's hammer, as can be clearly seen on the airflow UI.

First, create an SSH tunnel: ssh -NL 8778:an-airflow1001.eqiad.wmnet:8778 an-airflow1001.eqiad.wmnet

Then marvel at Mjolnir's graph view: http://localhost:8778/admin/airflow/graph?dag_id=mjolnir

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:Razzi/A_week_with_the_search_team&oldid=1957636"