Jump to content

OpenSearch Dashboards

From Wikitech
(Redirected from Kibana)

OpenSearch Dashboards (previously known as Kibana) is the frontend for Logstash, available at https://logstash.wikimedia.org.

This page is the user guide for OpenSearch Dashboards at WMF. For information about its operation, see Logstash. To read more about the software and its history, check OpenSearch on Wikipedia.

Quick intro

Where do logs come from?

Logs from MediaWiki end up here.

E.g. $logger = LoggerFactory::getInstance('Flow'); $logger->info(...) in MediaWiki PHP corresponds in Logstash to type:mediawiki channel:Flow level:INFO

For more about how to do that within works MediaWiki, see mw:Structured logging.

  • Start from one of the blue Dashboard links near the top, more are available from the Load icon near the top right.
  • In "Events over time" click and drag to zoom in to a specific region.
  • On the top right, you can change the time range to start with, e.g. last 24 hours, last 7 days. Smaller ranges are faster. If you find no results, check if you accidentally queried the future!
  • If you get lost, start again from the Home icon on the top right, or go to https://logstash.wikimedia.org/

See also slide 11 and onwards in the TechTalk on ELK by Bryan Davis, for more highlighted dashboard features (formerly known as "Logstash" or "Kibana").

Team dashboard and triage process talk

Watch "🎥 How to Logstash & Kibana (2020)" by Timo Tijhof (20 minutes, 📙 slides) to learn what production errors are, how we monitor production errors, and how you can create a team dashboard as part of a Phabricator triage workflow.

Beta Cluster Logstash

Web interface
https://beta-logs.wmcloud.org/
Access control
Credentials for Beta's Logstash can be found on officewiki, or by connecting to deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud and reading /root/secrets.txt. Unlike production services, Beta Cluster may not use Developer accounts (LDAP) for authentication.
$ ssh deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud -- sudo cat /root/secrets.txt
service: https://beta-logs.wmcloud.org
user: ************
password: ************
Kafka access to deployment-prep
The security group kafka-logging must allow ingress from the logging collector on port 9093.
When commissioning a new logging collector, the certificate authority keystore (/etc/ssl/localcerts/wmf-java-cacerts) must be manually copied onto the new logging collector otherwise logstash will not start: (File does not exist or cannot be opened /etc/ssl/localcerts/wmf-java-cacerts).

Tips

Discovery page

  • On the home page, click on "Discover: Run ad-hoc Logstash queries".
  • Select the index to search within. At the time of writing, the default selected index is "dlq-*". You may prefer to search in "logstash-*" for example.
  • Type some words within the search field. e.g., type "gitlab" to look for Gitlab logs.
  • Change the time range to increase the probability of your request returning something.

Homepage

The Home page is itself also dashboard. It has a single text panel with a Markdown list of links. Add your own for easy access!

Unfortunately it's not possible to just copy & paste stuff out of a search window (because formatting will break badly). If you feel like you really need the data, you can copy the POST request via firefox dev tools (as cURL) and then: 

# "log" being the field to print from each document
curl ... | gunzip | jq -r '.rawResponse | .hits | .hits | .[]._source | "\(.timestamp) \(.log)" | gsub("[\\n\\t]"; "")'

Gotchas

The browser address bar for Logstash navigations, are by default personalised to your login session. Sharing such link with other people, leads to a "Unable to restore URL" error.

To share results with others, use the "Share" link from the top right navigation, choose "Permalinks", enable "Short links", and then press "Copy link".

By default, shared links use the "Snapshot" mode which means it captures the state of the dashboard queries and panels as-is. This includes e.g. the timestamp slider, so if you're viewing "Last 1 hour" then the shared link will show different results an hour from now.

To share a specific result, use the single document link instead. Expand one of the raw events in the feed down on the dashboard, and copy the "View single document" link.

No results

  • If you see no events at all, perhaps you are querying the future only?
  • If you see no results or the results seem unrelated, press the magnifying glass at right of the main query bar to submit again. There is a race condition where if you modify the query while it is running, it ends up re-submitting the last completed query instead of the one you just wrote.
  • If you see events suddenly stop, perhaps the query includes the future (e.g. "Today" and "This week" instead of "Last 24h" or "Last week").
  • If you think you found when results first started to match your query, double check if it aligns with "today - 90 days ago" which is our message retention.

Visualisation panels

  • The visualisation panels are re-usable and thus saved globally.
  • Avoid changing existing visualisations unless intending to change other dashboards that use it at the same time. Otherwise "Save as.." under a new name first.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=OpenSearch_Dashboards&oldid=2234440"