OpenSearch Dashboards
OpenSearch Dashboards (previously known as Kibana) is the frontend for Logstash, available at https://logstash.wikimedia.org.
This page is the user guide for OpenSearch Dashboards at WMF. For information about its operation, see Logstash. To read more about the software and its history, check OpenSearch on Wikipedia.
Quick intro
- Logs from MediaWiki end up here.
- E.g.
$logger = LoggerFactory::getInstance('Flow'); $logger->info(...)in MediaWiki PHP corresponds in Logstash totype:mediawiki channel:Flow level:INFO - For more about how to do that within works MediaWiki, see mw:Structured logging.
- E.g.
- Start from one of the blue Dashboard links near the top, more are available from the Load icon near the top right.
- In "Events over time" click and drag to zoom in to a specific region.
- On the top right, you can change the time range to start with, e.g. last 24 hours, last 7 days. Smaller ranges are faster.
- If you get lost, start again from the homepage at https://logstash.wikimedia.org/
Team dashboard and triage process talk
- Learn how to Monitor production errors.
- MediaWiki team dashboard template: https://logstash.wikimedia.org/goto/216ae8c1a4837f54c58947a3294d0fb2
Watch "How to Logstash & Kibana (2020)" by Timo Tijhof (video, NDA-restricted, public slides) for a walkthrough on how to monitor production errors and how to create a team dashboard.
Tips
Homepage
The Home page is itself also dashboard. It has a single text panel with a Markdown list of links. Add your own for easy access!
Copy results out of a search
Unfortunately it's not possible to just copy & paste stuff out of a search window (because formatting will break badly). If you feel like you really need the data, you can copy the POST request via firefox dev tools (as cURL) and then:
# "log" being the field to print from each document
curl ... | gunzip | jq -r '.rawResponse | .hits | .hits | .[]._source | "\(.timestamp) \(.log)" | gsub("[\\n\\t]"; "")'
Gotchas
Link sharing
The browser address bar for Logstash navigations, are by default personalised to your login session. Sharing such link with other people, leads to a "Unable to restore URL" error.
To share results with others, use the "Share" link from the top right navigation, choose "Permalinks", enable "Short links", and then press "Copy link".
By default, shared links use the "Snapshot" mode which means it captures the state of the dashboard queries and panels as-is. This includes e.g. the timestamp slider, so if you're viewing "Last 1 hour" then the shared link will show different results an hour from now.
To share a specific result, use the single document link instead. Expand one of the raw events in the feed down on the dashboard, and copy the "View single document" link.
No results
- If you see no events at all, perhaps you are querying the future only?
- If you see no results or the results seem unrelated, press the magnifying glass at right of the main query bar to submit again. There is a race condition where if you modify the query while it is running, it ends up re-submitting the last completed query instead of the one you just wrote.
- If you see events suddenly stop, perhaps the query includes the future (e.g. "Today" and "This week" instead of "Last 24h" or "Last week").
- If you think you found when results first started to match your query, double check if it aligns with "today - 90 days ago" which is our message retention.
Visualisation panels
- The visualisation panels are re-usable and thus saved globally.
- Avoid changing existing visualisations unless intending to change other dashboards that use it at the same time. Otherwise "Save as.." under a new name first.