GitLab/Monitoring

From Wikitech


GitLab is monitored by Prometheus and Icinga.

Metrics and dashboards

Metrics of GitLab instances gitlab1001 and gitlab2001 are collected by Prometheus.[1] Multiple Grafana dashboards can be found in the folder GitLab.

Currently the following exporters are enabled:

Exporter Port Path
nginx 8060 /metrics
redis 9121 /metrics
postgresql 9187 /metrics
gitlab-workhorse 9229 /metrics
rails 8083 /metrics
sidekiq 8082 /metrics
gitlab-server 9168 /metrics
gitlay 9236 /metrics

Related Task: T275170

Alerting

Alerting happens with multiple Icinga checks. See the overview page. We mostly rely on the checks which come from the basic vm setup (like filesystem checks, puppet check). Additionally the public facing endpoints of production GitLab are checked as well[2]:

Host check
gitlab.wikimedia.org https and cert check
gitlab.wikimedia.org ssh check for git ssh daemon

Individual GitLab components (like Postgresql or Redis) are not monitored currently. There are no additional rules or alerts configured in Prometheus Alertmanager.

In the future when individual GitLab components are separated adapting the existing upstream alerts is recommended.

  1. https://gerrit.wikimedia.org/r/c/operations/puppet/+/707860
  2. https://gerrit.wikimedia.org/r/c/operations/puppet/+/708530