Jump to content

Citoid

From Wikitech

Citoid is a stateless Node.js service running in Kubernetes. It processes HTTP requests containing a URL, ISBN number or other identifiers/search terms by retrieving citation information from external sources with help from a local nodejs Zotero server. It responds with a JSON structure containing the citation information.

Updating

Citoid runs behind service-runner, and should be regularly updated in line with service-template-node

See: mw:Service-template-node/Updating

Deployment

These are directions for deploying citoid, a nodejs service. More detailed but more general directions for nodejs services are at Migrating_from_scap-helm#Code_deployment/configuration_changes.

Locate build candidate

From gerrit, locate the candidate build. PipelineBot will post a message with the build name, i.e.

PipelineBot
Mar 15 9:22 PM

Patch Set 4:

Wikimedia Pipeline
Image Build SUCCESS

IMAGE:
 docker-registry.discovery.wmnet/wikimedia/mediawiki-services-citoid

TAGS:
 2019-03-15-000-production, 950e3b4468f2f84d3bb23c33ba68d8a26725

'2019-03-15-000-production' is the name of the build.

Add change via gerrit

Pipeline bot

Pipeline bot automatically creates commits to the deployments repo with the subject line "citoid: pipeline bot promote".

  1. Find the commit by going to https://gerrit.wikimedia.org/r/q/topic:%22pipeline-promote%22+(status:open). If there is none labelled "citoid" in the list, then there are no new versions to merge.
  2. +2 the commit to merge it.

Manually (deprecated)

  1. Clone deployment-charts repo.
  2. cd helmfile.d/services/citoid
  3. vi values.yaml
main_app:
  image: wikimedia/mediawiki-services-citoid
  limits:
    cpu: 10
    memory: 4Gi
  liveness_probe:
    tcpSocket:
      port: 1970
  port: 1970
  requests:
    cpu: 200m
    memory: 200Mi
  version: 2019-01-17-114541-candidate-change-me
  1. Make a CR to change the version value in values.yaml and after a successful review, merge it.
  2. After merge, log into a deployment server, there is a cronjob (1 minute) that will update the /srv/deployment-charts directory with the contents from git.

Ssh into the deploy machine. 

 ssh deployment.eqiad.wmnet

citoid runs on two of the available server farms, codfw and eqiad. You can confirm this by doing

Change into the citoid directory. 

cd /srv/deployment-charts/helmfile.d/services/citoid

List the files in the directory. This is what you should see. 

> ls

helmfile.yaml values-codfw.yaml values-staging.yaml values.yaml

Staging

>myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ 

helmfile -e staging -i apply

Apply may take awhile. 

>helmfile -e staging status (to check status again)

To confirm the staging cluster works, try a request: 

curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=979%201029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'

These should successfully retrieve a JSON list of citations.

The following should return an error: 

curl -k --header 'Accept: application/json; charset=utf-8' 'https://staging.svc.eqiad.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fupload.wikimedia.org%2Fwikipedia%2Fcommons%2F9%2F98%2FColoring_page_for_Wikipedia_Day_2019_in_NYC.pdf'

Production

codfw cluster: 

>myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ 
helmfile -e codfw -i apply
helmfile -e codfw status

Verify citoid is running with a curl request: 

curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.codfw.wmnet:4003/api?format=mediawiki&search=979%201029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.codfw.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'

Repeat the same process for the eqiad cluster. 

>myusername@deploy1001:/srv/deployment-charts/helmfile.d/services/citoid$ 
helmfile -e eqiad -i apply
helmfile -e eqiad status

Then verify with a curl request: 

curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.eqiad.wmnet:4003/api?format=mediawiki&search=9791029801297'
curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.svc.eqiad.wmnet:4003/api?format=mediawiki&search=https%3A%2F%2Fjournals.plos.org%2Fploscompbiol%2Farticle%3Fid%3D10.1371%2Fjournal.pcbi.1002947'


curl for all servers: 

curl -k --header 'Accept: application/json; charset=utf-8' 'https://citoid.discovery.wmnet:4003/api?format=mediawiki&search=9791029801297'

Testing behind restbase

And finally, you can test the restbase implementation which should give the same results@

https://en.wikipedia.org/api/rest_v1/#!/Citation/getCitation

Graphing and Logs

For an overview of how your deploy is doing, you can look at overall stats and logs:

https://grafana.wikimedia.org/d/NJkCVermz/citoid?orgId=1&refresh=5m&from=now-30m&to=now

https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631

Probe

https://grafana.wikimedia.org/d/_77ik484k/openapi-swagger-endpoint-state?forceLogin&from=now-30m&orgId=1&to=now&var-site=codfw

Monitoring

Logs

Kibana dashboard: https://logstash.wikimedia.org/app/dashboards#/view/5eaf4e40-f6b6-11eb-85b7-9d1831ce7631

Use kibana to look at logs. However, if necessary, you can look at the raw logs from the cluster directory, 

source .hfenv 
kubectl logs -l app=citoid -c citoid-production

Stats

Stats are done currently with both Prometheus and statsd- in progress of converting statsd. Citoid/Prometheus for more.

See also

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Citoid&oldid=2167565"