Logstash/Tasks

From Wikitech
Jump to: navigation, search
  • Yes check.svg Done Install Logstash stack in labs instance : http://logstash.instance-proxy.wmflabs.org/
  • Yes check.svg Done Direct beta cluster udp2log traffic to Logstash : 95468
  • Yes check.svg Done Start conversation with Ops about production hardware : RT #6325
  • Yes check.svg Done Write more filter rules to parse various message formats
  • Get a syslog feed from beta
  • Yes check.svg Done Package logstash jar for deployment
  • Yes check.svg Done Make a puppet module
  • Yes check.svg Done Determine architecture for production deployment
    • What log forwarding methods should be used?
      • syslog, udp2log, redis
    • How many Logstash instances?
      • We will start with 3 nodes
    • HA strategy?
      • Varnish, built-in Elasticsearch HA, Redis. There is no HA for udp2log at the moment.
    • QoS terms?
      • Not mission critical at this point. This will need to be revisited as usage increases but it seems unlikely that this will ever become the system of record for log storage.
  • Implement Redis based transit of structured json log messages from MW

See also

Bugzilla Wikimedia/Logstash