Jump to content

Logstash/Tasks

From Wikitech
  • Yes Done Install Logstash stack in labs instance : http://logstash.instance-proxy.wmflabs.org/
  • Yes Done Direct beta cluster udp2log traffic to Logstash : 95468
  • Yes Done Start conversation with Ops about production hardware : RT #6325
  • Yes Done Write more filter rules to parse various message formats
  • Get a syslog feed from beta
  • Yes Done Package logstash jar for deployment
  • Yes Done Make a puppet module
  • Yes Done Determine architecture for production deployment
    • What log forwarding methods should be used?
      • syslog, udp2log, redis
    • How many Logstash instances?
      • We will start with 3 nodes
    • HA strategy?
      • Varnish, built-in Elasticsearch HA, Redis. There is no HA for udp2log at the moment.
    • QoS terms?
      • Not mission critical at this point. This will need to be revisited as usage increases but it seems unlikely that this will ever become the system of record for log storage.
  • Implement Redis based transit of structured json log messages from MW

See also