Draft:AQS

The Analytics Query Service, or AQS, is a mapping of REST-style endpoints to data stores. The system is used to deliver data to consumers like Toolforge tools, custom dashboards like Wikistats, mw:Special Pages, and more. The main purpose of AQS as a piece of infrastructure is to make it easy to serve data.

Creating a new Endpoint

Step by step, this section contains all you need to set up a new AQS 2.0 endpoint. Step 0 is making sure you have data loaded up in a data store that is considered best to serve it. To determine this, please talk to the Data teams (Data Persistence and Data Platform Engineering).

Step 1: Make a repository

TODO First:

task T352950 and task T352952: clean up service scaffold golang and decide whether to add a more aqs-specific version of it or make "AQS family" one of the options when scaffolding a new service

task T352953: document testing

decide what to refactor out of the individual repositories

configuration/configuration.go

cassandra.go

MAKEFILE

druid-schemas.yaml

env.go

healthz.go (part of scaffold)

decide on coding conventions (tabs standard in Go world)

linter

task T352949: move repositories to reflect ownership

task T352948: implement Hugh's idea to poll/read the latest Druid mediawiki history reduced snapshot

(in a nutshell: have an airflow job just run every hour and produce a config file somewhere central that the AQS services can read. Then the druid AQS services read that file at least every 24 hours and update their settings from it. This way, if a new snapshot is available, it will be automatically picked up without any deploy or restart of services needed.)

Once you settle on a name for your new repository, a scaffold will help you get started. An overview of what the scaffold creates: Draft:AQS/Repository_scaffold_overview. We suggest pushing your code to https://gitlab.wikimedia.org/repos/data-engineering/aqs/<<your-repository-name>>.

Step 2: Unit Test and Implement

Section on unit tests: how to write

Section on aqs-assist: how to implement a service and make your tests pass

Section on Druid and Cassandra utilities in aqs-assist

Section on

Step 3: Integration Tests

Integration tests with Cassandra

Integration tests with Druid

Note on using other data stores

Step 4: Document

Section on in-code documentation and the API spec

Section on on-wiki documentation (decide on format by refactoring docs for the Existing Endpoints below)

Step 5: Deploy

Maybe move detailed deployment instructions to a subpage Draft:AQS/Deploy, but cover everything:

routing and REST Gateway

loading data in data stores and Cassandra/Druid/other

deploying the code itself and the permissions required