Monitoring/DatasourceError

From Wikitech
Disabling DatasourceError for the alert

DatasourceError alerts are generated by Grafana in response to the Grafana alerting engine experiencing some error fetching data to evaluate the alert state.

To see what error is occurring, find the event in the Alerts overview Logstash Dashboard. A description of the alert is provided under the labels.Error field and a link to the alert is provided under the labels.generatorURL field.

These notifications are usually not actionable for alerts recipients, but indicate a period of time where the monitoring engine was unable to determine the state of the service.

If you do not wish to be notified of this gap in monitoring, you can disable this type of notification for your alert(s) like so:

  • Navigate to your alert rule configuration page
  • Under "3 Alert evaluation behavior" - "Configure no data and error handling" change "Alert state if execution error or timeout" to "OK"
  • Click "save" at the top of the page.

See also

  • task T345358 DatasourceError alerts emitted by jinxer are unhelpful
  • task T345362 DatasourceError grafana alerting error message database is locked