Jump to content

Tool:Early warning bot

From Wikitech
Toolforge tools
Early warning bot
Website https://earlywarningbot.toolforge.org/
Description Provide feedback on a patch as soon as any of its jobs fail.
Keywords quibble, bot, ci
Author(s) Kosta Harlantalk
Maintainer(s) Kosta Harlan (View all)
Source code https://gitlab.wikimedia.org/kharlan/earlywarningbot
License MIT License

What

Early warning bot comments and/or votes on a Gerrit change as soon as any of the jobs for the change fail.

Why

For MediaWiki core and extensions, some jobs (e.g. Selenium) can take over 15 minutes to complete. Meanwhile, static analysis jobs like phan, phpcs, eslint complete within minutes. It is nice to know as soon as job fails, so that you can make a fix without needing to wait ~20 minutes for all jobs to complete. Yes, you can watch Zuul, but receiving a notification via email/IRC can be nice.

How

Quibble 1.5.0 accepts a --reporting-url argument. In Wikimedia CI, this is configured to point to https://earlywarningbot.toolforge.org/. When a job fails, Quibble checks to see if there repository that triggered the jobs has a quibble.yaml file (documentation). If the file exists, Quibble POSTs information to the endpoint with the job name, a URL to the build failure, and whether the repository is configured to receive comments and/or votes (Verified -1) on build failure.

The bot is deployed to earlywarningbot.toolforge.org and uses an API key to validate that the request is from Quibble, it then posts on the Gerrit change. If should_comment: 1 is in the project's quibble.yaml, the bot will leave a comment on the patch indicating that a job failed, with a link to the build URL. If should_vote: 1 is set in the project's quibble.yaml, the bot will vote Verified: -1 on the patch. (This is what jenkins-bot will eventually do when all jobs are finished for that patch.)

Using with your project

Create a quibble.yaml file with the following contents and commit it to your repository.

quibble.yaml
earlywarning:
    should_vote: 1
    should_comment: 1

To switch off the integration with the bot, change the values to 0 or delete the file.

Known issues

  • If a build fails because of a flaky test, and the bot is configured to vote, then running "recheck" would still leave a Verified: -1 on the patch. This may or may not then interfere with a +2 on the patch. The workaround is to manually remove the Verified: -1 vote after submitting a recheck comment.

Deployment

  • Merge the merge request
  • SSH to toolforge
  • become earlywarningbot
  • Run ./build-and-release.sh
build-and-release.sh
#!/bin/sh
set -ex
cd earlywarningbot
git pull
jsub -N build -mem 2G -sync y -cwd cargo build --release
cd ~/
webservice restart

We have a webservice template:

service.template
backend: kubernetes
type: golang1.11
extra_args:
  - /data/project/earlywarningbot/start.sh

And start.sh contains credentials used for the bot:

start.sh
#!/bin/sh
ROCKET_ADDRESS=0.0.0.0 ROCKET_LOG_LEVEL=normal ROCKET_KEY={redacted} ROCKET_GERRIT_URL="https://gerrit.wikimedia.org" ROCKET_GERRIT_USERNAME="earlywarningbot" ROCKET_GERRIT_PASSWORD="{redacted}" ./earlywarningbot/target/release/earlywarning

Monitoring

  • kubectl get pods
  • kubectl logs -f earlywarningbot-59476d45c9-th9j5