SRE/Observability
|
SRE Observability - Monitoring and Logging (Prometheus/Grafana and ElasticSearch, plus some Kafka). The Observability team, or "o11y" for short, works across SRE and Technology to provide teams with tools, platforms and insights into how systems and services are performing. It leverages technologies such as Grafana, Kibana/Logstash, Prometheus, AlertManager and more. |
|---|
How we work.
Overview
The Observability team (often referred to as o11y) maintains several tools while curating and building a collaborative roadmap for the Wikimedia Foundation. Maintaining several of these work streams provide challenges as there is work that is adjacent, related, and directly assigned to the Observability team workboard but is not easily distinguishable from each other.
The purpose of this document is to describe:
- How the #o11y team does its work
- How other teams can request assistance or time on our roadmap
- How we will surface scheduling dependencies
- How we will use Phabricator to boost visibility
- What tools we intend to use
How to connect (and why)
- Reach out:
- IRC #wikimedia-observability connect -- Quick questions and conversations
- Email The SRE Observability Mailing List -- Announcements and general inquiries/coordination
- Phabricator tasks/tags: -- Preferred method of contact, and where the bulk of our work tracking happen (See diagram below)
- VictorOps -- For emergency/outage escalations
Office Hours
We also have office hours every week; if you wish to join or participate please reach out using one of the previous channels to receive the invitation. Office hours happen At the beginning of every team planning session. These happen every Wednesday at 3pm UTC on google hangouts. Reach out for the invitation.
How we Organize our Work
Work comes from multiple sources, but most requests should land in Phabricator. The #observability tag/project is the tag we use for all incoming work. Tasks will be one of six major states: inbox, backlog, scheduled, in progress, radar, done/closed.
- Inbox: this is the default state, all work lands in #observability inbox
- Backlog: work is accepted but unscheduled (we don't know when it will be worked on, but we know its a good thing to do)
- Scheduled: work is scheduled to do (aka prioritized) we know what needs to be done, who will do it and when it will be done, this will be manifested as a FY-Q# tag (FY2021/2022 Q1 milestone tag or similar), plus an inbox for that tag or " UP Next" for that milestone.
- In progress: this indicates that the work is in progress and actively being worked on.
- Radar: this is work we consider we want to be informed but will not be on our workboard this is only for the #observability component tag kept for visibility purposes.
- Done. this will be a relative state tagged in a done column under a specific milestone when it gets completed.
The Observability team grooms incoming tasks on a weekly basis normally during planning meetings on Wednesday at 8:00 AM Pacific. Some requests may receive an out-of-band prioritization effort.
This happens by reviewing the inbox for the #observability (component) workboard and the #sre-observability (group) workboard. (Inbox for a quarter or UP next)
From there the task or request should go through a quick prioritization of "done this quarter" i.e. time-sensitive, or backlog if the task is actionable for the Observability team. Otherwise, the task goes to radar or is blocked in the backlog if unable to move forward. Tasks that do not have enough information provided to the groom will receive a follow-up comment and remain unprioritized in the general backlog until enough information has been collected to effectively perform the task.
Phabricator Workflow
- Tasks go into the Observability intake tags (#observability, or component tags #icinga, #prometheus, #logstash)
- Tasks are then groomed by a member of the observability team and triaged for priority. Then the task is tagged the appropriate subcomponent (subproject) area; one of the four:
- observability-alerting
- observability-metrics
- observability-logging
- observability-tracing (not in active use yet)
- Each of these areas is a subproject and prioritized accordingly to manage a product roadmap for each of these main workstreams
- Tasks then get tagged into a milestone for execution (scheduling the work).
- Then tasks are either scheduled as up next or scheduled to a specific quarter
The process is there iterated on periodically (weekly,quarterly, yearly)
How we Plan our Roadmap
Roadmap planning will be a rolling 1+ year roadmap with the goal to have a list of tasks pre-groomed and prioritized periodically (quarterly).
There are 6 major work categories that drive efforts of the o11y team:
- Alerting
- Metrics
- Logs
- Tracing (future)
- Maintenance/Incidents
- Incident Management
The goal of this process is to:
- Drive each of these major workstreams
- Clearly set goals and deliverables
- Quantify effort and time investment per workstream, according to the level of investment that the organization is interested in per each individual initiative, and
- Allocate the adequate amount of time to each initiative.
Prioritization
This is both a scheduled and a continual effort in sizing up work and importance/impact of specific work streams. The team is employing a simple forced rank list of priorities that are fed from the intake process and groomed by the team. This effort in turn is then taken to a spreadsheet where these projects are scored for overall feel on value and capacity.
Order of presence for prioritization:
- Tier1 (high): incidents, security events, privacy concerns, PII in logs, unbreak now events
- Tier2 (medium): project work (OKR), outside requests, maintenances
- Tier3 (low): non critical maintenances
- Tier4 (lowest): work to be done when all other prioritized work is completed
Project Work (OKR)
All project work is prioritized and groomed beforehand. Overarching project tasks are created in Phabricator with subtasks, both of which will be tagged with a FY/ Quarter "milestone" to indicate scheduling for projects that span multiple quarters or years.
Maintenance (non-OKR)
Planned maintenance will follow the same workflow as regular project work, unplanned maintenance or requests will be groomed and prioritized based on urgency and severity.
Work Cadence Summary
| Activity | Frequency | Where |
| Intake Grooming + Prioritization | Weekly | o11y office hours |
| Planning (rolling roadmap) | Quarterly | OKR Meetings |
| Annual Planning | Yearly | TBD |