Wikimedia infrastructure

Wikimedia infrastructure

Data centers

• Locations:
  • Eqiad data center (Ashburn, USA)
  • Codfw data center (Dallas, USA)
  • Esams data center (Amsterdam, NL)
  • Ulsfo data center (San Francisco, USA)
  • Eqsin data center (Singapore, SG)
  • Drmrs data center (Marseille, FR)
• Data center operations
  • Common data center spec
  • Platform-specific doc
• Naming conventions

Networking

• Network design
• Peering
• Link aggregation
• LVS
• IP and AS allocations
• Network cheat sheet
• Network guidelines
• Network monitoring

Global traffic routing

• Global traffic routing
• Caching overview
• DNS
• HTTPS
• HAProxy
• Varnish
• Apache Traffic Server
• Traffic cache hardware
• HTTP timeouts
• Purged
• MediaWiki HTTP cache headers
• Kafka HTTP purging
• URI Path Normalization

MediaWiki

• Application servers
• PHP 7 and php-fpm
• Dumps
• Memcached
• Mcrouter runbook
• Nutcracker
• Envoy
• MediaWiki HTTP cache headers
• Redis
• Parser cache
• Statsd
• Kubernetes

Multimedia

• Thumbor
• Swift
• Media storage
• upload.wikimedia.org
• Media backups

Data Persistence

• MariaDB architecture
• MariaDB troubleshooting
• MariaDB monitoring
• MariaDB backups
• Database Runbooks
• Bacula
• Media backups
• Offsite Backups
• Backup Runbooks
• Orchestrator

Observability

• Logs
• MediaWiki UDP logging
• Logstash (backend)
• OpenSearch Dashboards (frontend)
• Prometheus
• Thanos
• Statsd
• Graphite
• Grafana
• Alertmanager

Search

• § Elastica
• § CirrusSearch
• § Job queue
• § Administration

Data Engineering

• Analytics Systems
• Hadoop Cluster
• Druid
• AQS
• Data Lake
• Event Platform

[]

Current infographics

For older or more detailed infographics, see Infographics.

eqiad

codfw

esams

ulsfo

eqsin

drmrs

Open the Template:ClusterMap page.

» Data centers

» MediaWiki (Webrequest flow, as of 2020)

» Memcached (Memcached flow, as of 2020)

Infrastructure overview

» Networking (Topology as of 2019)

» HTTP Caching (traffic flow from cache PoPs)