MariaDB/monitoring

Wikimedia infrastructure

Data centers

Networking

Global traffic routing

MediaWiki

Multimedia

Data Engineering

Search

Data Persistence

• MariaDB architecture
• MariaDB troubleshooting
• MariaDB monitoring
• MariaDB backups
• Database Runbooks
• Bacula
• Media backups
• Offsite Backups
• Backup Runbooks
• Orchestrator

Infra Foundations

Observability

Performance Team

[]

Icinga

Example of icinga checks

TBD

Metrics

Prometheus

Hosts to be monitored at prometheus are controlled by the instance and server inventory at Zarcillo (db1115). In order to update prometheus, hosts have to be inserted, updated or deleted from Zarcillo and then run /usr/local/sbin/mysqld_exporter_config.py on the prometheus hosts (e.g. prometheus1003 and prometheus1004 for eqiad). This scripts run automatically every 30 minutes to check for changes.

The standard mysqld-prometheus-exporter is used for most metrics.

Grafana

Connection anomaly detected through Prometheus metrics+Grafana

Relevant dashboards:

• MySQL aggregated: https://grafana.wikimedia.org/d/000000278/mysql-aggregated
• MySQL instance: https://grafana.wikimedia.org/d/000000273/mysql
• Replication lag: https://grafana.wikimedia.org/d/000000303/mysql-replication-lag
• MediaWiki LoadBalancer: https://grafana.wikimedia.org/d/G9kbQdRVz/mediawiki-loadbalancer

Orchestrator

Orchestrator

Logstash/Kibana

Mediawiki database errors

TBD

This page is a part of the SRE Data Persistence technical documentation
(go here for a list of all our pages)