httpbb

From Wikitech

httpbb (for "HTTP Black Box") is a black-box testing tool for HTTP servers. It reads in a list of test cases, each of which is a specification for an HTTP request followed by one or more assertions about the response. Those requests are played against one or more servers, and the test passes if all the assertions come true.

The code is on Gerrit in the operations/software/httpbb repository.

Running tests

httpbb is installed on the deployment hosts and the cluster management hosts (cumin1002.eqiad.wmnet, cumin2002.codfw.wmnet); the standard tests for app servers are installed into /srv/deployment/httpbb-tests. To run, pass one or more YAML test files (described below) on the command line, and one or more servers with the --hosts flag. For example, to run the full set of tests against all four mwdebug hosts, 

rzl@cumin1001:~$ httpbb /srv/deployment/httpbb-tests/appserver/* --hosts=mwdebug[1001,1002].eqiad.wmnet,mwdebug[2001,2002].codfw.wmnet

(Tip: httpbb tests multiple hosts in parallel, but runs significantly faster when all hosts are in the local datacenter, due to the roundtrip latency for each request.)

Adding tests

The input to httpbb is one or more YAML files full of test cases. The app server tests are stored in modules/profile/files/httpbb in the Puppet repository.

The YAML has the following structure:

  • Each top-level key consists of http:// or https:// followed by a host. The hostname given is moved into the Host header for outgoing requests. Multiple host names may be used in the same file, corresponding to virtual hosts on the same server. The actual destination for the traffic is given by the --hosts command-line flag.
  • For each host key, the value is an array of test cases, expressed as mappings.
  • Each test case includes a path field (e.g. /wiki/Main_Page) for the remainder of the outgoing request URL.
  • Optionally, test cases may include a method (string). The default is GET.
  • Optionally, test cases may include a request_headers (string mapping). The default is the Host header as described above, and an httpbb User-Agent. Test cases may override the User-Agent header.
  • Optionally, test cases may include either a form_body (string mapping) or a json_body (any JSON-representable type) to set a request body, respectively either URL-encoded like POST-submitted form data or encoded as JSON. If present, httpbb encodes the contents and adds the appropriate Content-Type header automatically. The HTTP method is not set automatically -- if you're using form_body or json_body, you may also want to say method: POST. Setting both fields is invalid.
  • Each test case includes one or more assertion fields. If any assertion fails, the test fails. Supported assertions are:
    • assert_status (int) to check the status code of the response.
    • assert_headers (string mapping) to check one or more response headers. Header names are case-insensitive, and values must match exactly.
    • assert_headers_regex (string mapping) to check one or more response headers. Header names are case-insensitive, and values are partial-match regular expressions.
    • assert_headers_absent (string array) to assert that none of these (case-insensitive) header names are present in the response.
    • assert_body_contains (string) to check that a substring is present in the response text.
    • assert_body_regex (string) to check that the response text matches a multiline partial-match regular expression.

Examples

To assert that a server correctly serves the enwiki main page (or, at least, something with the correct title, along with an OK status): 

https://en.wikipedia.org:
- path: /wiki/Main_Page
  assert_status: 200
  assert_body_contains: Wikipedia, the free encyclopedia

To assert that the default path / on Wiktionary serves a redirect to the main page, and that main page is served: 

https://en.wiktionary.org:
- path: /
  assert_status: 301
  assert_headers:
    Location: https://en.wiktionary.org/wiki/Wiktionary:Main_Page
- path: /wiki/Wiktionary:Main_Page
  assert_status: 200
  assert_body_contains: Wiktionary, the free dictionary

(Since these tests share a virtual host, we don't spell out the key twice. This is a YAML mapping, so repeating a key would overwrite the previous value.)

Deploying httpbb

Follow this procedure to release a new version of httpbb.

Find out the current version, e.g. by checking the Tags list at gitiles. As of this writing, httpbb is sub-v1.0, and every new release just increments the patch version, so versions are numbered 0.0.1, 0.0.2, 0.0.3, etc. Throughout these instructions, replace "X.Y.Z" with your new version number.

Optionally, first open a Phabricator task ("Release httpbb version X.Y.Z") and self-assign it.

On your laptop:

  • In your clone of the httpbb repository, check out the tip of the master branch, or wherever you'll cut the release.
  • export VERSION=X.Y.Z (use your new version number)
  • git tag ${VERSION?} && git push gerrit ${VERSION?}
  • dch -v${VERSION?}-1 -Dbuster-wikimedia --force-distribution --urgency=low
    • This opens your editor and adds a new entry to the top of the Debian changelog. Include a brief description of what's new in this version.
    • If you see "dch warning: neither DEBEMAIL nor EMAIL environment variable is set," note that dch uses this email address in the changelog. You can either manually set your email address in the changelog, or (permanent solution) cancel with Ctrl-C, set $EMAIL in your ~/.bashrc, ~/.zshrc, or other shell config file, restart your shell, and then retry.
  • git commit -a
    • Use a commit message like "Release vX.Y.Z." Include a "Bug:" header for your release task, if appropriate.
  • Review and merge the commit as usual.

On build2001.codfw.wmnet:

  • export VERSION=X.Y.Z (use your new version number)
  • mkdir -p build-area/httpbb
  • If you don't already have the repo checked out on the build host, git clone https://gerrit.wikimedia.org/r/operations/software/httpbb
    • This is an anonymous clone (over https) to avoid messing around with SSH keys; we won't need to push from the build host anyway.
  • cd httpbb
  • If you didn't just create the clone, ensure your working directory is clean and then git pull
  • First, build for buster: WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=buster --git-export-dir=../build-area/httpbb --git-upstream-tag=${VERSION?}
  • Now, rebuild for bullseye: dch --edit
    • In this case you can dismiss the "neither DEBEMAIL nor EMAIL environment variable is set" warning, since your email address isn't needed at this step.
    • This opens the Debian changelog in your editor. In the first line, change (X.Y.Z-1) to (X.Y.Z-1+deb11u1) and change buster-wikimedia; to bullseye-wikimedia. Save and exit.
  • Commit this change with git commit -am "Rebuild for bullseye" but keep it local to the build host; don't send it to gerrit.
  • WIKIMEDIA=yes gbp buildpackage -sa -us -uc --git-pbuilder --git-no-pbuilder-autoconf --git-dist=bullseye --git-export-dir=../build-area/httpbb --git-upstream-tag=${VERSION?}

Don't forget that you still have that extra commit in your httpbb repository on build2001. When you're done with the release, you can get rid of it with git checkout -B master origin/master, so that it doesn't get in your way next time.

On apt1001.wikimedia.org:

  • export VERSION=X.Y.Z (use your new version number)
  • mkdir -p httpbb/buster httpbb/bullseye
  • rsync rsync://build2001.codfw.wmnet/pbuilder-result/buster-amd64/*httpbb* httpbb/buster
  • sudo -i reprepro -C main include buster-wikimedia ${HOME}/httpbb/buster/httpbb_${VERSION?}-1_amd64.changes
  • In #wikimedia-operations connect, !log the previous command and include your tracking task number.
  • rsync rsync://build2001.codfw.wmnet/pbuilder-result/bullseye-amd64/*httpbb* httpbb/bullseye
  • sudo -i reprepro -C main include bullseye-wikimedia ${HOME}/httpbb/bullseye/httpbb_${VERSION?}-1+deb11u1_amd64.changes
  • In #wikimedia-operations connect, !log that command too, and include your tracking task number.

On a cumin host:

  • generate-debdeploy-spec
    • This will interactively walk you through generating a debdeploy spec.
    • The name of the source package is httpbb.
    • No comment is needed, so leave it blank.
    • The update type is tool.
    • The version fixed in bullseye is your X.Y.Z-1+deb11u1.
    • The version fixed in buster is your X.Y.Z-1.
    • We don't build httpbb for stretch, so leave it blank.
    • No binary packages are being renamed, so leave it blank.
  • If it's been less than 30 minutes since you ran reprepro include, run sudo cumin C:httpbb 'apt update'
  • Using the spec file that generate-debdeploy-spec wrote, sudo debdeploy deploy -u ${SPEC_FILE?} -Q C:httpbb

Done! Close your release task, if appropriate.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Httpbb&oldid=2050421"