Help:Toolforge/Alerts
If you maintain one or more Toolforge tools, you can opt in to receive notification emails when certain error conditions are detected on one of your tools.
Which problems are detected
Toolforge checks every tool for these problems:
| Alert | When it fires |
|---|---|
| ToolWebserviceDown | For more than 20 minutes, more than half of the requests to the web service of the tool get no response: the web service may be stopped, crashing, or unreachable. |
| ToolWebserviceErrors | For more than 20 minutes, the web service of the tool answers more than half of its requests with a server error (HTTP 5xx). The web service is running, but it has an application problem. |
| ToolFrequentRestarts | The containers of the tool restarted more than 5 times in total in the last hour. This usually means that a process is crashing, or that it runs out of memory. It covers both the web service and the jobs of the tool. |
What you receive
- The emails are sent to
your-tool.alerts@toolforge.org. By default, this address reaches all the current maintainers of the tool. To customize who receives the emails, create a.forward.alertsfile in the home directory of the tool: see Help:Toolforge/Email#Mail to a Tool. - Tool maintainers receive one email when an alert starts, one email every 24 hours while it continues, and one when it is resolved.
- Each email names the alert, says since when it has been firing, explains what is happening, and suggests what to check. It links to the page of your tool in toolsadmin.
- The emails are sent from
noreply@toolforge.org. You cannot reply to this address.
Enabling the email notifications
The setting is part of the Toolforge configuration of your tool. Add this to your toolforge.yaml file:
defaults:
alerts:
notify: true
Then store the new configuration:
$ toolforge components config create toolforge.yaml
You do not need to deploy your tool again: the setting applies from the next check, within a minute or two. To see the current setting, run toolforge components config show.
The setting applies to the whole tool, and all its maintainers receive the emails.
Disabling the email notifications
You can just remove the lines added before (as the default is to be disabled at the time of writing this), or set explicitly notify to false in your toolforge.yaml file, and store the configuration again:
defaults:
alerts:
notify: false
...
$ toolforge components config create toolforge.yaml
Every email also tells you how to turn them off.
What to do when you receive a Toolforge alert email
ToolWebserviceDown and ToolWebserviceErrors
- Open your tool and see how it answers.
- Read the logs of your web service.
- Check that it is running.
- Restart it if it is stuck.
TODO: add common debugging commands.
See Help:Toolforge/Web for more information.
ToolFrequentRestarts
- Find the job or web service that restarts, and read its logs to look for the error that stops it.
- If the logs show no error, check that the process keeps running: a job or web service that exits is restarted, even when it exits without an error.
- If the process runs out of memory, give it a larger memory limit.
TODO: add common debugging commands.
See Help:Toolforge/Running jobs for more information.
Listing alerts
There is currently no interface to list the active alerts for a tool. We plan to add a CLI and UI for listing alerts in the future, but there is no timeline yet.
Known limitations
- A stopped web service does not alert. If you stop the web service on purpose, no alert fires: the alerts are for tools that try to run and fail.
- A slow web service does not alert. Only failed requests and errors count.
- A web service that receives no traffic does not alert, unless it keeps restarting.
- The restarts of all the containers of a tool add up. A few restarts in several jobs can fire the restart alert, even if no single job is in trouble. This includes jobs where restarts are expected, such as the runs of a scheduled job.
- The alerting service can miss an alert while it is down itself. If an alert starts while the alerting service restarts, and the restart takes more than a minute, you may not receive the first email for it.
Related links
- Admin page for the Alerting service that implements this feature.
Feedback
We would like to know how this feature works for you: the alerts that were useful, the ones that were not, and the problems it missed.
You can use our usual communication channels, listed below.
Communication and support
Support and administration of the WMCS resources is provided by Wikimedia Foundation staff and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect or the bridged Telegram group
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read posts about Cloud Services on the Wikimedia Technical Blog. (Older posts can be read from the Cloud Services Blog.)