Wikifunctions/Runbook
Jump to navigation
Jump to search
This is a list of runbooks for the Abstract Wikipedia Team, particularly the Wikifunctions services, covering step-by-step lists of what to do when things need doing, especially when things go wrong.
How to disable execution in an emergency
Prevent logged-out users from running functions
- In mediawiki-config, change core-Permissions.php to set
'wikilambda-execute' => false
in the'*'
(logged-out users) block ofgroupOverrides => '+wikifunctions'
, push to gerrit, and get deployed like any MW config change.
Prevent all users from running functions
- In mediawiki-config, change core-Permissions.php to set
'wikilambda-execute' => false
in the'*'
(logged-out users) and also the'user'
(logged-in users) blocks ofgroupOverrides => '+wikifunctions'
, push to gerrit, and get deployed like any MW config change.
Back-end services
- Main article: Kubernetes/Deployments#Deploying_with_helmfile
What services?
function-orchestrator
, a service to co-ordinate function requests.function-evaluator
, a service to execute user-written code.
Deploy a config update to the orchestrator
- Make a change to the Wikifunctions services helm values over-ride in the deployment-charts repo in gerrit, make a commit, land it with a colleague or by yourself
- Shell into production deployment server (
ssh deployment.eqiad.wmnet
) and go to our service directory (cd /srv/deployment-charts/helmfile.d/services/wikifunctions
) - Check that the new change to deployment-charts git repo has made it automatically to the server (
git log
) - Run
helmfile -e staging -i apply --context 5
to validate that the helm chart applies and the diff looks correct. If it shows no diff, wait a little for the chart museum to catch up, then try again. - Make a simple request via curl to check that the orchestrator performs as expected, e.g.:
curl https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate -X POST --data '{"zobject":{"Z1K1":"Z7","Z7K1":"Z801","Z801K1":"foo"},"doValidate":false}' --header "Content-type: application/json" -w "\n"
- … should output a JSON blob starting with
{"Z1K1":"Z22","Z22K1":"foo",…
(call just to the orchestrator)curl https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate/ -X POST --data '{"zobject":{ "Z1K1": "Z7", "Z7K1": { "Z1K1": "Z8", "Z8K1": [ "Z17", { "Z1K1": "Z17", "Z17K1": "Z6", "Z17K2": { "Z1K1": "Z6", "Z6K1": "Z400K1" }, "Z17K3": { "Z1K1": "Z12", "Z12K1": [ "Z11" ] } }, { "Z1K1": "Z17", "Z17K1": "Z6", "Z17K2": { "Z1K1": "Z6", "Z6K1": "Z400K2" }, "Z17K3": { "Z1K1": "Z12", "Z12K1": [ "Z11" ] } } ], "Z8K2": "Z1", "Z8K3": [ "Z20" ], "Z8K4": [ "Z14", { "Z1K1": "Z14", "Z14K1": "Z400", "Z14K3": { "Z1K1": "Z16", "Z16K1": { "Z1K1": "Z61", "Z61K1": "python" }, "Z16K2": "def Z400(Z400K1, Z400K2):\n\treturn str(int(Z400K1) + int(Z400K2))" } } ], "Z8K5": "Z400" }, "Z400K1": "5", "Z400K2": "8" },"doValidate":false}' --header "Content-type: application/json" -w "\n"
- … should output a JSON blob starting with
{"Z1K1":"Z22","Z22K1":"13",…
(call to the Python evaluator via the orchestrator)curl https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate/ -X POST --data '{"zobject":{ "Z1K1": "Z7", "Z7K1": { "Z1K1": "Z8", "Z8K1": [ "Z17", { "Z1K1": "Z17", "Z17K1": "Z6", "Z17K2": { "Z1K1": "Z6", "Z6K1": "Z400K1" }, "Z17K3": { "Z1K1": "Z12", "Z12K1": [ "Z11" ] } }, { "Z1K1": "Z17", "Z17K1": "Z6", "Z17K2": { "Z1K1": "Z6", "Z6K1": "Z400K2" }, "Z17K3": { "Z1K1": "Z12", "Z12K1": [ "Z11" ] } } ], "Z8K2": "Z1", "Z8K3": [ "Z20" ], "Z8K4": [ "Z14", { "Z1K1": "Z14", "Z14K1": "Z400", "Z14K3": { "Z1K1": "Z16", "Z16K1": "Z600", "Z16K2": "function Z400( Z400K1, Z400K2 ) { return (parseInt(Z400K1) + parseInt(Z400K2)).toString(); }" } } ], "Z8K5": "Z400" }, "Z400K1": "15", "Z400K2": "18" },"doValidate":false}' --header "Content-type: application/json" -w "\n"
- … should output a JSON blob starting with
{"Z1K1":"Z22","Z22K1":"33",…
(call to the JavaScript evaluator via the orchestrator)curl https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate/ -X POST --data '{"zobject":{"Z1K1":"Z7","Z7K1":"Z10000","Z10000K1":"foo","Z10000K2":"bar"},"doValidate":false}' --header "Content-type: application/json" -w "\n"
- … should output a JSON blob starting with
{"Z1K1":"Z22","Z22K1":"foobar",…
(call to the evaluator via the orchestrator, making a call to the wiki)
- Run
helmfile -e codfw -i apply --context 5
to deploy the update to the Texas datacentre (the change is now live for some users) - Run
helmfile -e eqiad -i apply --context 5
to deploy the update to the Virginia datacentre (the change is now live for all users) - Monitor production for a bit, and revert if needed
- Wikifunctions services grafana dashboard
- For the above
curl
commands, you can replacewikifunctions.k8s-staging.discovery.wmnet
withwikifunctions.discovery.wmnet
.
Deploy a new version of the orchestrator
- Make a change to the orchestrator repo in GitLab, make a Merge Request, wait for it to landed by a colleague (example)
- Make a config change to the orchestrator's helm chart as above, changing in
values-main-orchestrator.yaml
theversion
value of the docker image to the newly-created docker-registry tag from step 1. You may wish to explain in the commit what changes are being deployed, for ease of tracking later. (example)
Disable an evaluator from being called
- Make a config change to the orchestrator's helm values as above, changing in
values-main-orchestrator.yaml
theORCHESTRATOR_CONFIG
value to remove the evaluator from the map of known evaluators.- If the evaluator you are removing is the only one assigned to that language, you are disabling evaluation in that language.
Add an evaluator to be called
- To add a new evaluator instance, edit
helmfile.yaml
to add a new entry in thereleases
section, and add a newvalues-foo-evaluator.yaml
file like the others but pointed at the appropriate image and version. Deploy this, and ensure the new release deploys successfully. - Make a config change to the orchestrator's helm chart as above, changing in
values-main-orchestrator.yaml
theORCHESTRATOR_CONFIG
value to add the evaluator to the map of known evaluators for the appropriate languages.- If the evaluator you are adding is the only one assigned to that language, remember that you are enabling evaluation in that language.
Deploy a config update update to an evaluator
Note: There is intentionally very little configurability of the evaluators themselves.
- Identify which evaluator release you're updating (JavaScript, Python, etc.) or if it's for all evaluators.
- Find the
values-*-evaluator.yaml
file for the release you wish to adjust, and alter them to add/update/remove the relevant config values, and deploy as above (example)
Deploy a new version of an evaluator
- Make a change to the evaluator repo in GitLab, make a Merge Request, wait for it to landed by a colleague (example)
- Make a config change to the Wikifunctions service helm values as above, changing in one or all of the
values-*-evaluator.yaml
files theversion
value of the docker image to the newly-created docker-registry tag from step 1. You may wish to explain in the commit what changes are being deployed, for ease of tracking later. (example)
Wikifunctions.org wiki
Add a new pre-defined Object in production
- Shell into a maintenance server
- If you only need to add one Object, run:
mwscript extensions/WikiLambda/maintenance/loadPreDefinedObject.php --wiki=wikifunctionswiki --zid <NUMERICAL ZID>
, e.g.mwscript extensions/WikiLambda/maintenance/loadPreDefinedObject.php --wiki=wikifunctionswiki --zid 1234
- If you have a few to add, run:
mwscript extensions/WikiLambda/maintenance/loadPreDefinedObject.php --wiki=wikifunctionswiki --from <NUMERICAL ZID> --to <NUMERICAL ZID>
, e.g.mwscript extensions/WikiLambda/maintenance/loadPreDefinedObject.php --wiki=wikifunctionswiki --from 1234 --to 1237
How to monitor usage
This section is currently a draft. Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page. |
- Wikifunctions service general dashboard (WIP)
- Envoy telemetry usage dashboard of the function-orchestrator
How to debug an issue
This section is currently a draft. Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page. |
How to inspect Wikifunctions with Kubernetes tools
- Log in to a deployment server and run
kube_env wikifunctions $CLUSTER
where$CLUSTER
is probablystaging
unless you are very brave. - Now you can run
kubectl
commands likekubectl get pods
. - For example, to get logs for the function-orchestrator, you can run
kubectl logs `kubectl get pods | grep orchestrator | awk '{print $1}'` function-orchestrator-main-orchestrator
.
How to poke the orchestrator/evaluator
- Log in to a deployment server.
- The endpoint for the orchestrator can be found at
https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate/
. - Example commands:
- to provoke the orchestrator directly:
curl https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate/ -X POST --data '{"zobject":{"Z1K1":"Z7","Z7K1":"Z801","Z801K1":"foo"},"doValidate":false}' --header "Content-type: application/json"
- to call the evaluator via the orchestrator:
curl https://wikifunctions.k8s-staging.discovery.wmnet:30443/1/v1/evaluate/ -X POST --data '{"zobject":{ "Z1K1": "Z7", "Z7K1": { "Z1K1": "Z8", "Z8K1": [ "Z17", { "Z1K1": "Z17", "Z17K1": "Z6", "Z17K2": { "Z1K1": "Z6", "Z6K1": "Z400K1" }, "Z17K3": { "Z1K1": "Z12", "Z12K1": [ "Z11" ] } }, { "Z1K1": "Z17", "Z17K1": "Z6", "Z17K2": { "Z1K1": "Z6", "Z6K1": "Z400K2" }, "Z17K3": { "Z1K1": "Z12", "Z12K1": [ "Z11" ] } } ], "Z8K2": "Z1", "Z8K3": [ "Z20" ], "Z8K4": [ "Z14", { "Z1K1": "Z14", "Z14K1": "Z400", "Z14K3": { "Z1K1": "Z16", "Z16K1": { "Z1K1": "Z61", "Z61K1": "javascript" }, "Z16K2": "function Z400( Z400K1, Z400K2 ) { return (parseInt(Z400K1) + parseInt(Z400K2)).toString(); }" } } ], "Z8K5": "Z400" }, "Z400K1": "5", "Z400K2": "8" } ,"doValidate":false}' --header "Content-type: application/json"
- to provoke the orchestrator directly:
Thing that might go wrong
- environment variables
- Environment variables set in the microservice images are ignored by Kubernetes. If you add/delete/modify an environment variable in a container image, you must also update the corresponding configuration when deploying that version of the image.