User:Majavah/Loki notes

This page is currently a draft.
More information and discussion about changes to this draft on the talk page.

This page collects some draft notes from investigating and testing Grafana Loki as a potential solution for Toolforge log aggregation

Collection

promtail sidecar, with an emptyDir to log to
- TESTED (toolsbeta.test3): this works fine at least with uwsgi, interface will be "send your logs to this file"
no formatting requirements! just ship your plain text
custom docker image with promtail and a script to create the full config file with relevant labels (at least container name, possibly deployment name as well), credentials mounted from a k8s secret
in a production deployment this would probably have a kubernetes mutating webhook to mount the sidecar based on a k8s annotation
- proof of concept

loki server
ngx_http_auth_request_module and a custom microservice for auth + x-scope-orgid header

loki does not have native authentication built-in, relies on some proxy for that (but is multi-tenant, and expects the proxy to set a header)
promtail (official ingestion agent) supports http basic auth / client tls
we can probably write some code to provision some of those auth methods for each tool and write them to a kubernetes secret
- TESTED: proof of concept works!

grafana is standard
- grafana is natively multi-tenant (organizations), and data sources are per org
- supports proxy auth but no account autocreation / support for reading groups from header
Grafana + CAS + custom script to create orgs? can possibly re-use parts of the script used to create grafana.wikimedia.org users from ldap