Performance/Runbook/Arc Lamp service
This is the runbook for deploying and monitoring Arc Lamp. Arc Lamp collects stack traces with php-excimer from MediaWiki and publishes flame graphs and trace logs.
Arc Lamp data originates from the Arc Lamp client (configured in
operations/mediawiki-config.git:/Profiler.php) that runs alongside MediaWiki on the application servers. The samples are collected with php-excimer, and sent to a Redis instance on the mwlog host.
The Arc Lamp service then reads the stream of samples from Redis, and produces trace logs on disk. The flame graphs are generated with brendangregg/FlameGraph.
Stats from the Arc Lamp client are sent to Graphite. Stats from the Arc Lamp service go to Prometheus.
For how to migrate to a new host, refer to past arclamp1001 upgrade (T319434).
- Source code: https://gerrit.wikimedia.org/g/performance/arc-lamp
- Server provisioning: modules/arclamp and profile::webperf::arclamp.
The puppet role for these services is role::webperf:profiling_tools. We currently run one
arclamp#### host in each core data center (Eqiad and Codfw). You can monitor these via the Grafana host overview dashboard. There's also an equivalent deployment-webperf## host in the Beta Cluster.
- Grafana dashboard: Arc Lamp
- Index page: https://performance.wikimedia.org/php-profiling/
- Public file listing: https://performance.wikimedia.org/arclamp/svgs/daily/
- Upstream brendangregg/FlameGraph.