Caching overview

From Wikitech
Jump to: navigation, search
Wikimedia infrastructure

Data centres and PoPs

Networking

HTTP Caching


Wiki

Media

Logs

Graph

 __________________
| browser/the webz |
|__________________|
    |
    |
    |
  ____________________
 |  LVS               |
 |    (load balancer) |
 |____________________|
     |
     |
     |
     |
    __________________________________________________________
   |  Edge Frontend (Varnish)                                 |
   |     Short-lived cache (~10sec, mostly to prevent DDOS)   |
   |     Stored in memory                                     |
   |__________________________________________________________|
         |
         |
         |
       _______________________________
      | Edge Backend (Varnish)        |
      |  Long-lived cache             |
      |  Stored on disk               |
      |_______________________________|
            |
            |
            |
           _______________________________________
          | Apaches (MediaWiki PHP)               |
          |                                       |
          |   * Cache-Control for page view HTML: |
          |      max-age is 14 days               |
          |   * wikitext parsercache:             |
          |      Expires at 30 days               |
          |_______________________________________|

Varnish

  • Frontends max ttl is 1 day.
  • Backends max ttl is 14 days.

MediaWiki

Invalidating content

For Varnish:

  • When pages are edited, their canonical url is proactively purged in Varnish by MediaWiki.

For ParserCache: Values in ParserCache are verifiable by revision ID. Edits will naturally update it.

  • puppet: manifests/misc/maintenance.pp
    • class misc::maintenance::parsercachepurging
      • Set to 30 days (expire age=2592000)

Past events

  • 2013: Prevent white-washing of expired page-view HTML.
    • Various static aspects of a page are not tracked or versions, as such, when the max-age expires, a If-Not-Modified must not return true after expiry even if the database entry of the wiki page was unchanged.
    • More info: https://phabricator.wikimedia.org/T46570
  • 2016: Decrease max object ttl in Varnish