News/2023 Cloud VPS metrics changes
This page details information about the changes to Cloud VPS metrics systems in mid 2023.
What is changing?
Relevant changes that happened previously
- Debian 11 (released mid 2021) no longer includes Diamond, which produced machine-level stats for Graphite.
- From November 2022, a Prometheus instance (at prometheus.wmcloud.org) has been collecting machine-level statistics using prometheus-node-exporter on all VMs.
What is changing now?
- Monday, July 17th: The remaining Cloud VPS instances will have Diamond removed. This means Graphite-based per-instance metrics will stop working.
Done - Tuesday, August 1st
- The Cloud VPS Graphite and statsd servers will be shut down. Done
- The Grafana server at grafana-cloud.wikimedia.org and grafana-labs.wikimedia.org will be shut down and replaced by a new server running on grafana.wmcloud.org. Done
- The Cloud VPS Graphite and statsd servers will be shut down.
What should I do?
Migrate dashboards (and other tools) to use Prometheus data
You should migrate any affected Grafana dashboards and other tools you find useful to load data from Prometheus instead of loading it from Graphite.
If your project needs Graphite for a reason other than per-machine statistics and can't move to Prometheus, you need to run your own Graphite instance instead of relying on the WMCS-hosted instance.
Migrate dashboards from grafana-cloud.wikimedia.org to grafana.wmcloud.org
You should move any Grafana dashboards you find useful on grafana-cloud.wikimedia.org to grafana.wmcloud.org.
There is a dump of all of the dashboards that existed on grafana-cloud before its shutdown.
Why are we doing this?
- We are replacing Diamond metrics with Prometheus based ones because Diamond is written in Python 2, which is no longer supported on latest versions of the Debian operating system.
- We are shutting down the Graphite server because it's primary purpose was to collect Diamond data.
- We are replacing the old Grafana instance because new versions of Grafana no longer supports the setup we currently have.
See also
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)