Citoid
Citoid is a stateless Node.js service running in Kubernetes. It processes HTTP requests containing a URL, ISBN number or other identifiers/search terms by retrieving citation information from external sources with help from a local nodejs Zotero server. It responds with a JSON structure containing the citation information.
Updating
Citoid runs behind service-runner, and should be regularly updated in line with service-template-node
See: mw:Service-template-node/Updating
Deployment
These are directions for deploying citoid, a nodejs service. More detailed but more general directions for nodejs services are at Migrating_from_scap-helm#Code_deployment/configuration_changes.
Locate build candidate
From gerrit, locate the candidate build. PipelineBot will post a message with the build name, i.e.
PipelineBot Mar 15 9:22 PM Patch Set 4: Wikimedia Pipeline Image Build SUCCESS IMAGE: docker-registry.discovery.wmnet/wikimedia/mediawiki-services-citoid TAGS: 2019-03-15-000-production, 950e3b4468f2f84d3bb23c33ba68d8a26725
'2019-03-15-000-production' is the name of the build.
Add change via gerrit
Pipeline bot
Pipeline bot automatically creates commits to the deployments repo with the subject line "citoid: pipeline bot promote".
- Find the commit by going to https://gerrit.wikimedia.org/r/q/topic:%22pipeline-promote%22+(status:open). If there is none labelled "citoid" in the list, then there are no new versions to merge.
- +2 the commit to merge it.
Manually
- Clone deployment-charts repo.
- cd helmfile.d/services/citoid
Update image
- vi values.yaml
main_app: image: wikimedia/mediawiki-services-citoid limits: cpu: 10 memory: 4Gi liveness_probe: tcpSocket: port: 1970 port: 1970 requests: cpu: 200m memory: 200Mi version: 2019-01-17-114541-candidate-change-me
- Make a CR to change the version value in values.yaml and after a successful review, merge it.
- After merge, log into a deployment server, there is a cronjob (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
Config change
Increment chart version in: charts/citoid/Chart.yaml
Changes values in the template: charts/citoid/templates/_config.yaml
Navigating the server
Ssh into the deploy machine.
ssh deployment.eqiad.wmnet
citoid runs on two of the available server farms, codfw and eqiad. You can confirm this by doing
Change into the citoid directory.
cd /srv/deployment-charts/helmfile.d/services/citoid
List the files in the directory. This is what you should see.
> ls
helmfile.yaml values-codfw.yaml values-staging.yaml values.yaml
Staging
>myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ helmfile -e staging -i apply
Apply may take awhile.
>helmfile -e staging status (to check status again)
To confirm the staging cluster works, try a request:
Production API
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/PMC3605911'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/979%201029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fwww.wsj.com%2Fworld%2Fasia%2Fhistoric-moon-mission-gets-china-one-small-step-ahead-in-space-race-41894d41'
These should successfully retrieve a JSON list of citations.
The following should return an error:
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F9%2F98%2FColoring_page_for_Wikipedia_Day_2019_in_NYC.pdf'
Deprecated API
Should all return:
{"error":"Api endpoint unavailable as of 2.0.0.; use path parameters instead."}
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=979%201029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fwww.wsj.com%2Fworld%2Fasia%2Fhistoric-moon-mission-gets-china-one-small-step-ahead-in-space-race-41894d41'
Production
codfw cluster:
>myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ helmfile -e codfw -i apply helmfile -e codfw status
Verify citoid is running with a curl request:
curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.codfw.wmnet:4003/mediawiki/979%201029801297' curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.codfw.wmnet:4003/mediawiki/https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'
Repeat the same process for the eqiad cluster.
>myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ helmfile -e eqiad -i apply helmfile -e eqiad status
Then verify with a curl request:
curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.eqiad.wmnet:4003/mediawiki/9791029801297' curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947' curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.eqiad.wmnet:4003/mediawiki/https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F9%2F98%2FColoring_page_for_Wikipedia_Day_2019_in_NYC.pdf'
curl for all servers:
curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.discovery.wmnet:4003/mediawiki/9791029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.discovery.wmnet:4003/mediawiki/https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F9%2F98%2FColoring_page_for_Wikipedia_Day_2019_in_NYC.pdf'
Testing behind REST Gateway
And finally, you can test the REST gateway implementation which should give the same results@
https://en.wikipedia.org/api/rest_v1/#!/Citation/getCitation
REST gateway citoid grafana board
Graphing and Logs
For an overview of how your deploy is doing, you can look at overall stats and logs:
https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=now-30m&to=now
https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631
Pyrra
Citoid availability SLO:
The citoid grafana dashboard is based on the logs and logs only get logged if the service is able to do so. The citoid availability metric in pyrra checks for 5xx. 500s get logged, but 503s (service not responding) where the service is not up at all do not, so is good to check here as an uptick in 503s will not show up elsewhere.
Full SLO board: https://slo.wikimedia.org/?search=citoid
Quarterly SLO objectives: https://grafana-rw.wikimedia.org/d/YuUMRZ44z/slo-quarterly-review?forceLogin=true&from=2025-06-01T00:00:00.000Z&orgId=1&timezone=utc&to=2025-08-31T23:59:59.000Z
Monitoring
- Icinga: Search for 'citoid'
- Grafana shows probe history for all swagger probes, including for citoid: https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?orgId=1. Logs for swagger endpoint alerts: https://logstash.wikimedia.org/app/dashboards#/view/8b1907c0-2062-11ec-85b7-9d1831ce7631?_g=h@faedd66&_a=h@c160d35
Logs
Kibana dashboard: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631
Use kibana to look at logs. However, if necessary, you can look at the raw logs from the cluster directory,
kube_env citoid eqiad # or codfw kubectl logs -l app=citoid -c citoid-production
Stats
Stats are done currently with Prometheus. Citoid/Prometheus for more.