Test Kitchen/Test Kitchen UI/Administration
Test Kitchen UI is an instrument and experiment management system. The application enables product teams to set up, manage, and configure various instruments for data collection and experimentation. This includes scheduling, designating metadata, setting sample rates, and targeting deployment environments.
Service details
| Attribute | Value |
|---|---|
| Owner | Experiment Platform |
| Kubernetes Cluster | dse-k8s-eqiad |
| Kubernetes Namespace | test-kitchen |
| Chart | https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/charts/test-kitchen/ |
| Helmfiles | https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/helmfile.d/dse-k8s-services/test-kitchen |
| Docker image | https://docker-registry.wikimedia.org/repos/data-engineering/test-kitchen/tags |
| Internal service DNS | test-kitchen.svc.eqiad.wmnet |
| Public service URL | https://test-kitchen.wikimedia.org |
| Logs | https://logstash.wikimedia.org/app/dashboards#/view/d1587980-dd85-11ef-ae49-21d4ac0f91a0 |
| Metrics | https://grafana-rw.wikimedia.org/d/ee2057f3-eb34-45a7-a48b-489e3ff0b2ec/test-kitchen-service?var-service=test-kitchen |
| Monitors | TBD |
| Application documentation | https://wikitech.wikimedia.org/wiki/Test_Kitchen/Test_Kitchen_UI |
| Paging | false |
| Deployment Phabricator ticket | https://phabricator.wikimedia.org/T361335 |
Deployment
When to deploy
Deployments to staging environment (https://test-kitchen-next.wikimedia.org) can be done at will to test new features:
- Tasks in review will remain there after being merged and they will be moved to "Sign Off" to be deployed to staging, to be tested there
- Once those tasks have been tested on staging, they will be moved to "To Deploy"
- Once a task that is in "To Deploy" is deployed to production, it can be move to "Done"
Deployments to production environment (https://test-kitchen.wikimedia.org) will be done in a controlled way:
- There is a Test Kitchen UI deployment window, set every Tuesday at 14:00h UTC (7:00 PDT)
How to deploy
Note that:
test-kitchen-nextis the staging environmenttest-kitchenis the production environment
# Do the following once you have prepared the change and the change has been reviewed and merged
ssh deployment.eqiad.wmnet
cd /srv/deployment-charts
git log -n 1
# Check that the change has been pulled
cd helmfile.d/dse-k8s-services/<environment>
# Check changes and deploy
helmfile -e dse-k8s-eqiad diff
helmfile -e dse-k8s-eqiad -i apply
Troubleshooting
See Test Kitchen/Troubleshooting for some important tips or Kubernetes/Troubleshooting for a comprehensive guide about troubleshooting Kubernetes
Additionally, note that:
test-kitchen-nextis the staging environmenttest-kitchenis the production environmenttest-kitchen-stagingis the container name for staging environment (test-kitchen-next)test-kitchen-productionis the container name for production environment (test-kitchen)
Check pods
$ ssh deployment.eqiad.wmnet
$ kube_env <environment> dse-k8s-eqiad
$ kubectl get pods -w
NAME READY STATUS RESTARTS AGE
test-kitchen-staging-57d4bbb6bf-z2p5k 1/2 Error 0 10s
test-kitchen-staging-57d4bbb6bf-z2p5k 1/2 Error 1 (1s ago) 10s
test-kitchen-staging-57d4bbb6bf-z2p5k 1/2 CrashLoopBackOff 1 (1s ago) 11s
See logs
$ ssh deployment.eqiad.wmnet
$ kube_env <environment> dse-k8s-eqiad
$ kubectl get pods -w
$ kubectl logs -f <pod_name> -c <container_name> --tail=10
{"level":50,"time":1714483575453,"pid":1,"hostname":"test-kitchen-staging-66554859c4-p9m28","msg":"Error while setting an env variable. There is no value for SAL_PASSWORD"}
{"level":30,"time":1714483575592,"pid":1,"hostname":"test-kitchen-staging-66554859c4-p9m28","msg":"Connected to the database at an-mariadb1001.eqiad.wmnet:test_kitchen_staging"}
{"level":30,"time":1714483575600,"pid":1,"hostname":"test-kitchen-staging-66554859c4-p9m28","msg":"test-kitchen has started listening on port 8080"}
You can also see logs in their corresponding Logstash dashboards:
Troubleshoot the deployment
Let's assume we want to troubleshoot the deployment of a new release in staging environment:
$ ssh deployment.eqiad.wmnet
$ kube-env test-kitchen-next-deploy dse-k8s-eqiad
$ kubectl get events
Troubleshoot wmftkbot
See the related toolforge continuous job
Aditionally, we could need to check
$ ssh login.toolforge.org
$ become tk
# we could check first if the job is running
$ toolforge jobs list
# we could also need to see all the information about the job
$ toolforge jobs show tk-sal-logger
Stop/delete the job
Let's assume wmftkbot is malfunctioning and, for example, flooding the SAL, and we need to stop/delete the related toolforge job:
$ ssh login.toolforge.org
$ become tk
$ toolforge jobs delete tk-sal-logger
Create/start the job
After having to stop/delete the toolforge job, at some point we will need to create/start it again:
$ ssh login.toolforge.org
$ become tk
$ toolforge jobs run tk-sal-logger --command="cd test-kitchen/toolforge && pyvenv/bin/python sal_logger.py" --image python3.13 --continuous
Troubleshoot maintenance jobs
Known issues
We know that, every so often, a phabricator ticket with the title MediaWikiCronJobFailed is filed to alert that a Test Kitchen maintenance job has failed. The recurrent issue is Failed to load configuration from etcd: which is completely unrelated to Test Kitchen and there is nothing to do on our side. We were said that it's something that happens from time to time and we don't have to be worried about it. In this case we can remove safely that failed job.
Look for failed jobs
Let's assume we have received an error about a maintenance job that has failed. We could look for failed jobs and, once we have found the relevant one, we could take a look at its logs to see what happenned
$ ssh deployment.eqiad.wmnet
$ kube_env mw-cron eqiad
$ kubectl get jobs --field-selector status.successful=0 | grep testkitchen-updateconfigs
$ kubectl logs jobs/testkitchen-updateconfigs-29577414 mediawiki-main-app
Manually delete a failed job
Once we have troubleshot the failing job, we may want to delete it
$ ssh deployment.eqiad.wmnet
$ kube_env mw-cron eqiad
$ kubectl get jobs --field-selector status.successful=0 | grep testkitchen-updateconfigs
$ kubectl delete job testkitchen-updateconfigs-29577414