Introduction / motivation

In parallel with moving services to kubernetes (k8s) we're also moving services away from statsd metrics and onto Prometheus instead. For more information on Prometheus see also https://wikitech.wikimedia.org/wiki/Prometheus and https://phabricator.wikimedia.org/T205870

Not all services support Prometheus out of the box; as a compatibility layer between Prometheus and statsd we're providing statsd_exporter (https://github.com/prometheus/statsd_exporter): a daemon to receive statsd metrics, map them onto Prometheus metrics and expose the result on HTTP for Prometheus to ingest.

The conversion done by statsd_exporter is driven by a set of mappings that will need to be provided as part of the service's helm chart. This document provides guidelines and best practices for service owners on how to write said mappings.

One of the biggest differences between statsd and Prometheus is the data model: statsd/graphite have a flat hierarchy derived by the metric name, whereas Prometheus associates multiple key/value pairs to a single metric name. See also this guide for more information on the data model: https://prometheus.io/docs/concepts/data_model/

Choosing metric names and tags

In WMF's deployment we're following Prometheus' naming best practices as close as possible, thus https://prometheus.io/docs/practices/naming/ is a very good read to get started on what's in a Prometheus metric name and how to best pick metric names for services.

Global aggregation considerations

One of the big paradygm shifts between statsd and Prometheus has to do with metric aggregation across hosts and service instances: at the moment the way aggregation works is that services and hosts send their statsd traffic to statsd.eqiad.wmnet and aggregation is performed centrally.

For services running in k8s and statsd_exporter the aggregation is done at the single service instance level; in other words services send their statsd traffic to localhost for statsd_exporter to turn that into Prometheus metrics. The resulting metrics are then periodically fetched (scraped) by Prometheus. Moving aggregation to service instances works most of the time for most metric data types (see also https://prometheus.io/docs/concepts/metric_types/) though notably summaries (e.g. percentiles/quantiles of how long an operation took) cannot be aggregated in a statistically meaningful way across different service instances. A similar (aggregatable) data type is provided in the form of histograms: each timing observation is put into "count of observations that took less than" buckets, and percentiles on aggregated values can be calculated server-side by Prometheus. See also https://prometheus.io/docs/practices/histograms/ for a more detail explanation of summaries and histograms.

Example mappings

The statsd_exporter mappings configuration below provides an example common to all service-runner based services: namely gc and heap stats. The mappings can be extended to add service-specific metrics (e.g. request latency)

mappings:
  - match: '*.gc.*'
    name: 'service_runner_gc_${2}_nanoseconds'
    timer_type: histogram
    buckets: [ 5e+5, 1e+6, 5e+6, 10e+6, 15e+6, 30e+6, 50e+6 ]
    labels:
      service: '$1'
  # workaround, service-runner sends heap stats as timers ATM 
  - match: '*.heap.*'
    name: 'service_runner_heap_${2}_bytes'
    timer_type: histogram
    buckets: [1e+6, 1e+7, 1e+8, 1e+9]
    labels:
      service: '$1'

The resulting metrics e.g. for mathoid look something like this (modulo heap stats that should be a gauge not a timer):

# curl -s localhost:9112/metrics | grep -i service
# HELP service_runner_gc_minor_nanoseconds Metric autogenerated by statsd_exporter.
# TYPE service_runner_gc_minor_nanoseconds histogram
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="500000"} 15
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="1e+06"} 15
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="5e+06"} 15
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="1e+07"} 15
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="1.5e+07"} 15
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="3e+07"} 15
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="5e+07"} 15
service_runner_gc_minor_nanoseconds_bucket{service="mathoid",le="+Inf"} 15
service_runner_gc_minor_nanoseconds_sum{service="mathoid"} 19873.929999999997
service_runner_gc_minor_nanoseconds_count{service="mathoid"} 15
# HELP service_runner_heap_rss_bytes Metric autogenerated by statsd_exporter.
# TYPE service_runner_heap_rss_bytes histogram
service_runner_heap_rss_bytes_bucket{service="mathoid",le="1e+07"} 1
service_runner_heap_rss_bytes_bucket{service="mathoid",le="1e+08"} 1
service_runner_heap_rss_bytes_bucket{service="mathoid",le="1e+09"} 1
service_runner_heap_rss_bytes_bucket{service="mathoid",le="+Inf"} 1
service_runner_heap_rss_bytes_sum{service="mathoid"} 132775.936
service_runner_heap_rss_bytes_count{service="mathoid"} 1
# HELP service_runner_heap_total_bytes Metric autogenerated by statsd_exporter.
# TYPE service_runner_heap_total_bytes histogram
service_runner_heap_total_bytes_bucket{service="mathoid",le="1e+07"} 1
service_runner_heap_total_bytes_bucket{service="mathoid",le="1e+08"} 1
service_runner_heap_total_bytes_bucket{service="mathoid",le="1e+09"} 1
service_runner_heap_total_bytes_bucket{service="mathoid",le="+Inf"} 1
service_runner_heap_total_bytes_sum{service="mathoid"} 98381.824
service_runner_heap_total_bytes_count{service="mathoid"} 1
# HELP service_runner_heap_used_bytes Metric autogenerated by statsd_exporter.
# TYPE service_runner_heap_used_bytes histogram
service_runner_heap_used_bytes_bucket{service="mathoid",le="1e+07"} 1
service_runner_heap_used_bytes_bucket{service="mathoid",le="1e+08"} 1
service_runner_heap_used_bytes_bucket{service="mathoid",le="1e+09"} 1
service_runner_heap_used_bytes_bucket{service="mathoid",le="+Inf"} 1
service_runner_heap_used_bytes_sum{service="mathoid"} 85099.288
service_runner_heap_used_bytes_count{service="mathoid"} 1