Search Platform/Weekly Updates/2025-04-17
Highlights
Shipped: did we release anything this week?
- Nothing significant this week
Blockers: does this essential workstream have any unresolved blockers or dependencies? is anything preventing us from doing our work?
- None
Lessons learned: Did we learn anything in the course of doing this work that can be applied to other work?
- Vector Search PoC completed. Read the conclusion on ticket, but some highlights:
- retrieval speed is very interesting compared to MoreLike
- quality is hard to judge, we think MoreLike is overall better, but we should run an A/B test to validate
- memory usage seems high, might be difficult to scale
Community collab: Did we do anything in this essential workstream this week in collaboration with the community or because the community asked us to?
- N/A
Details
Vector Search PoC
MLR Improvements
- Updating mjolnir airflow orchestration (which is the first we ever wrote) to work like everything else we have in airflow. Simplifies the DAG and brings years of airflow learnings to our most complicated DAG. - https://phabricator.wikimedia.org/T390061
Misc / Operations
- Adjusted fixture handling in the shared airflow support code to improve fixture diffs for all users: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1239
- Update airflow test suite to prevent users from deleteing a fixture-supported aspect of the repo without also deleting the related fixture (keeps the repo clean): https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/1240
- Restored mjolnir-msearch daemon to operating in all datacenters after the issue with the opensearch 1.x LTR plugin mis-handling missing values was addressed by patching the plugin and rolling out a new version. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1136414
- T390853 Consider using upgradeMode=savepoint for the cirrus-streaming-updater
- T390853 Consider using upgradeMode=savepoint for the cirrus-streaming-updater
- T326311 Deletion of Lexemes appears to leak triples related to its forms and senses
- T221709 scap service restarts for WDQS are inconsistent
- T389429 Investigate whether it’s intentional / correct that default CirrusSearch setups run cirrusSearchElasticaWrite as separate jobs
- T391232 drop_mjolnir_partitions is broken