Search Platform/Weekly Updates/2023-04-28

From Wikitech

Summary

Annual planning took a reasonable amount of mind share this week. Search Update Pipeline is moving forward. Some unexpected work around operations and development support.

What we've accomplished

Search Analysis

Search Update Pipeline

  • Write a small job that copies the cirrus index data in avro to a smaller dataset written in parquet, will be used to identify inconsistencies between the cirrus index the mysql page table.
  • Upgrade WDQS Streaming Updater to Flink 1.16, testing is done, deployment to production is still pending - https://phabricator.wikimedia.org/T289836

Search SLO

Operations / SRE

  • Hardware requests for next fiscal year

Misc

Wrote a small patch to cleanup error handling of CirrusSearch writes (work initially started by Emmanuel).

  • Tried to figure out what is going on with WCQS/commons next year, if anything.
  • Met with Growth team to talk about product collaboration between reader/editor needs and search, disc, brows. There is a lot of shared thinking and overlap, and hopefully there's some room to collaborate with the team on reader discovery stuff.