User:DChan (WMF)/EditSuggestionSLO
Status: initial draft. Nothing here is guaranteed to be accurate.
Organizational
Service
In the Edit Check framework, within a VisualEditor edit session, edit suggestions are checks that run on the entire article (not just content the user has modified). In other words, edit suggestions are checks running in suggestion mode. Edit suggestions can check all kinds of aspects of the article content: authorial tone, formatting, spelling conventions, appropriate linking, likely LLM output, etc. Edit suggestions can run:
- entirely within the browser; or
- partly on a backend server (e.g. querying an AI language model, as Tone Check does)
Some edit suggestions are highly configurable on each individual wiki. For instance, TextMatchEditCheck can be given custom rules, effectively allowing wiki administrators to create their own checks (some of which could have a significant performance burden).
Teams
The Editing team is responsible the development and maintenance of the Edit Check codebase including many edit suggestions.
Individual wiki communities are responsible for deciding which edit suggestions to deploy, including configuration
The Site Reliability Engineering team is responsible for maintaining Edit Check instances running on production infrastructure.
The Release Engineering team is responsible for deploying updates to Edit Check on production infrastructure.
Architectural
Note about performance: computational demand
From a computation standpoint, checking the entire article is far more demanding than just checking content the user has modified. Roughly speaking, check running time gets multipied by three potentially large numbers:
- amount of content — long articles may contain hundreds of paragraphs.
- re-run frequency — Checks results may become stale every time the article is modified (i.e. potentially on every keystroke), forcing frequent re-runs
- number of checks — There may be dozens of checks operating.
The computational demand can be reduced with optimization techniques such as caching, scoping and asynchronicity. However these impose a cost in terms of complexity. Therefore it is important to have an SLO, to decide how much optimization work is appropriate. We wish to monitor performance, not availability.
Environmental dependencies
VisualEditor, running on MediaWiki, is the specific platform within which Edit Suggestions operate.
Browser performance greatly constrains Edit Suggestions performance, because running checks is computationally intensive. Timings on a low-end mobile device may be orders of magnitude slower than on a modern desktop.
Browser ecosystem issues could prevent edit suggestions from displaying or operating correctly; this should be excluded from SLIs.
Community-editable wiki configuration could greatly affect performance; this should be included in SLIs.
Service dependencies
Some checks make HTTP calls to backend services, e.g. AI language models.
Client-facing
Clients
Feature users: end-user editors, wiki administrators / patrollers
Secondary clients
API users: teams who are writing checks (Growth, Moderation Tools), gadget authors on-wiki
Request Classes
[TODO]
Service Level Indicators (SLIs)
Latency SLI: This measures the lag between initiating a run of checks and displaying settled results. [TODO: should probably be based on the latency stats collected by EditCheckPerformance (and logged to server). And we should probably somehow focus the results on the slowest devices. And big articles.]
Input responsiveness SLI: This measures the extent to which the synchronous portion of the checking process slows the keyboard responsiveness. [TODO: details, again using EditCheckPerformance]
Availability SLI: None
Operational
Monitoring
[TODO]
Troubleshooting
An SLI failure could be due either to changed code or changed configuration. Troubleshooting the former would mean a code fix, deployed either via the train or as a backport. Troubleshooting the latter would mean a direct change in on-wiki configuration (e.g. disabling a TextMatchEditCheck rule that proves too computationally intensive).
Deployment
Deployment is via the train (or backports), or via community configuration.
Service Level Objectives
Realistic targets (reasonable worst case)
Fixing an SLI failure might require a code change, which could take a day to notice as it rolls out on the train, and Editing could fix it within a day. If one of these happens each quarter, then it shouldn’t decrease the SLI by more than one percentage point.
On the other hand, an SLI failure might be due to configuration, which could be noticed and fixed more quickly but might happen more often. Again we can suppose it might burn a day per quarter.
This means an SLO of 98% is realistic.
Ideal targets (reasonable best case)
An SLI failure is probably most severe for a slow mobile device editing a very long page: only in this case would it render VisualEditor near-unusable (until the user switches off suggestions). Slow mobile device + very long page is a fairly unusual combination (say <2.5%). So assuming a future VisualEditor SLO of something like 99.95%, an SLO of 98% would still be appropriate.