Jump to content

Arc Lamp

From Wikitech

This is the internal runbook for deploying and monitoring Arc Lamp. Arc Lamp collects stack traces from all MediaWiki production traffic, using php-excimer, and publishes daily aggregated flame graphs and trace logs to https://performance.wikimedia.org/php-profiling/.

Service

Arc Lamp data originates from the Arc Lamp client (configured in operations/mediawiki-config.git:/Profiler.php) that runs alongside MediaWiki on the application servers. The samples are collected with php-excimer, and sent to a Redis instance on the arclamp host.

The Arc Lamp service then reads the stream of samples from Redis, and produces trace logs on disk. The flame graphs are generated with brendangregg/FlameGraph.

Stats from the Arc Lamp client are sent to Graphite. Stats from the Arc Lamp service go to Prometheus.

Architecture of Arc Lamp (as of December 2019).

Provision

We currently run one arclamp#### host in each core data center (Eqiad and Codfw). There's also an equivalent deployment-webperf## host in the Beta Cluster.

Monitoring

Host migration

For how to migrate to a new host, refer to past arclamp1001 upgrade (T319434).

See also