ORES/WMF details

From Wikitech
Jump to navigation Jump to search

These are notes specific to the WMF deployment of ORES.

Redis

We're using two redundant Redis nodes in each data center. (Master-replica, or clustered?) Data centers don't exchange any Redis data or scores.

We've enabled Redis's builtin LRU cache expiration to automatically manage memory.

Note that our score caching is not critical to API capacity, it's just a courtesy to users with repetitive workloads. For example, patrollers using a tool like Huggle will often load the most recent 100 changes, which is a near-instantaneous operation thanks to caching, rather than having to score 100 revisions before responding. The contribution of our score cache to reducing overall server load is only about 2%, see these graphs: https://grafana.wikimedia.org/dashboard/db/ores?orgId=1&from=1496275200000&to=1512086400000

Dictionaries

Some feature extraction modules require aspell or compatible dictionary packages for each language. We maintain a port of the Indonesian dictionary, here: https://apt.wikimedia.org/wikimedia/pool/main/a/aspell-id/