Performance/Synthetic testing/Result storage/Runbook

This is the runbook for the result storage for synthetic testing.

Meta

• Issue tracker (Phabricator): synthetic-performance-testing
• Documentation: ResultStorage

Debug

If results for the tests is missing or not working you will see that in Grafana by either that the "see test result" annotation do now show screenshots when you hover the metrics or when you to go to the result of a test and you get a 404.

If all results are missing there is probably a problem with nginx serving the data. If there's just one or a couple of tests that miss data, it could have problem uploading the data.

Uploading problems

Servers that upload content use the user uploader that has access to the uploading folders. You can find if there's been problem uploading files by checking the Slack channel #synthetic-tests-error-reporting

If there's been problem with uploading you will see logs looking something like this:

hetzner-1-baremetal [2024-10-15 04:24:49] ERROR: Error: Connection lost before handshake

hetzner-1-baremetal [2024-10-15 04:24:49] ERROR: Could not upload using scp Error: Connection lost before handshake

If that happens a lot you need to dig into why SCP has problems. If it happens a couple of times for month you can just ignore it.

Serving content problems

The nginx server is straightforward. If you think something is wrong with nginx you can check the status using:

sudo systemctl status nginx

If you the server Is down you can start it with sudo systemctl start nginx

If you need to change the configuration you reload it with sudo systemctl reload nginx