User:Ottomata/MediaWiki State
I'm trying to rephrase some of what is stated in Shared_Data_Platform into a more focused problem and use cases, rather than the broad theory (Data as a Product) and platform (Common Data Infrastructure) language used there.
If you are reading this, please feel free to make edits!
Externalized MediaWiki state problem
MediaWiki data is needed outside of MediaWiki:
- Search indexes
- ML model training
- WDQS
- caches
- WM Enterprise
- Dumps
- MediaWiki History
- wikistats
- Cloud VPS Wiki Replicas and other public Data Services
- backups
- ‘right to fork’
- etc.
all need externalized MediaWiki state.
The current systems for getting this data suffer from two main problems:
- No way to bootstrap the full state. Dumps 1.0 sort of helps, but its complicated.
- No way to consistently get updates to state. MW state change events are best effort only. There are missed events.
This problem has never been holistically considered or addressed together as an org. Until we do:
- Engineers using MediaWiki data outside of MediaWiki will continue to spend resources resolving these problems as best they can for their individual projects, wasting time and resources that could be better spent focused on building product features (see list above).
- New product ideas that require external state may be abandoned because of the difficulty to build them.
See also
- T291120 - MediaWiki Event Carried State Transfer - Problem Statement describes this problem in more detail.
- T120242 - Consistent MediaWiki state change events is about solving 2.
- Dumps 2.0 will hopefully help with 1., but it is not clear that it is meant to do so (it is meant to replace the aging Dumps 1.0 system).
- Shared_Data_Platform talks about this problem more theoretically (Data as a Product, not just MediaWiki etc.).