Logstash/Tasks

From Wikitech
Jump to navigation Jump to search
  • Done Install Logstash stack in labs instance : http://logstash.instance-proxy.wmflabs.org/
  • Done Direct beta cluster udp2log traffic to Logstash : 95468
  • Done Start conversation with Ops about production hardware : RT #6325
  • Done Write more filter rules to parse various message formats
  • Get a syslog feed from beta
  • Done Package logstash jar for deployment
  • Done Make a puppet module
  • Done Determine architecture for production deployment
    • What log forwarding methods should be used?
      • syslog, udp2log, redis
    • How many Logstash instances?
      • We will start with 3 nodes
    • HA strategy?
      • Varnish, built-in Elasticsearch HA, Redis. There is no HA for udp2log at the moment.
    • QoS terms?
      • Not mission critical at this point. This will need to be revisited as usage increases but it seems unlikely that this will ever become the system of record for log storage.
  • Implement Redis based transit of structured json log messages from MW

See also

Bugzilla Wikimedia/Logstash