User:Taavi/Loki notes

This page is currently a draft.
Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page.

This page collects some draft notes from investigating and testing Grafana Loki as a potential solution for Toolforge log aggregation

Collection

promtail sidecar, with an emptyDir to log to
- TESTED (toolsbeta.test3): this works fine at least with uwsgi, interface will be "send your logs to this file"
no formatting requirements! just ship your plain text
custom docker image with promtail and a script to create the full config file with relevant labels (at least container name, possibly deployment name as well), credentials mounted from a k8s secret
in a production deployment this would probably have a kubernetes mutating webhook to mount the sidecar based on a k8s annotation
- proof of concept

Ingestion

loki server
ngx_http_auth_request_module and a custom microservice for auth + x-scope-orgid header

Authentication

loki does not have native authentication built-in, relies on some proxy for that (but is multi-tenant, and expects the proxy to set a header)
promtail (official ingestion agent) supports http basic auth / client tls
we can probably write some code to provision some of those auth methods for each tool and write them to a kubernetes secret
- TESTED: proof of concept works!

Storage

loki has code for swift, but does not list as supported in official docs?
no support for scaling local file storage unlike OpenSearch
need to estimate storage needs

Querying

grafana is standard
- grafana is natively multi-tenant (organizations), and data sources are per org
- supports proxy auth but no account autocreation / support for reading groups from header
Grafana + CAS + custom script to create orgs? can possibly re-use parts of the script used to create grafana.wikimedia.org users from ldap

Other

Debian packages for Loki/Promtail? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001903