Fundraising/tech/Alerts

Service alerts

As of May 2026, fr-tech is using prometheus to manage service alerts. The alerts are generated from metrics within the frack infrastructure and sent to the production alerts hosts. The production infrastructure controls the routing for notifications and alerting.

In general, alerts fired from prometheus notify via email, IRC, and if critical Splunk/VictorOps.

• IRC alerts are announced in #wikimedia-fundraising by the bot jinxer-wm.
• Email alerts are generally sent to fr-tech-ops@wm.o with certain alerts going to fr-tech@wm.o.
• Alerts that are critical and page / wake someone up worthy are routed to Splunk/VictorOps where they will be delivered to the fr-tech-ops members cell phones.

Alerts are created and managed in the frack puppet repo at modules/prometheus/files/rules The yaml files in that directory are deployed to the frmon hosts and used for crafting and triggering alerts.

Managing alert notifications

Alert notification pathways are controlled in the production puppet repo. The routes and receivers can be found in alertmanager.yml.erb

All alerts generated from frack prometheus are tagged with a team label of fr-tech or fr-tech-ops:

    labels:
        team: 'fr-tech'

This label determines the notification methods to use for alerting.

Viewing alerts

The alerts dashboard is implemented using Karma, and can be reached at: https://alerts.wikimedia.org/

Silencing alerts

General documentation for silencing alerts can be found at Alertmanager#Silences_&_acknowledgements

Custom silences could also be created by using the Bell icon in the upper left of the alerts dashboard.

Additional documentation

Additional documentation on using Karma can be found at:

• Karma docs from Caascad - https://docs.caascad.com/en/alerting/karma.html
• Karma dashboard - https://karma-dashboard.io/