Logstash/Tasks

Done Install Logstash stack in labs instance : http://logstash.instance-proxy.wmflabs.org/
Done Direct beta cluster udp2log traffic to Logstash : 95468
Done Start conversation with Ops about production hardware : RT #6325
Done Write more filter rules to parse various message formats
Get a syslog feed from beta
Done Package logstash jar for deployment
Done Make a puppet module
Done Determine architecture for production deployment
- What log forwarding methods should be used?
  - syslog, udp2log, redis
- How many Logstash instances?
  - We will start with 3 nodes
- HA strategy?
  - Varnish, built-in Elasticsearch HA, Redis. There is no HA for udp2log at the moment.
- QoS terms?
  - Not mission critical at this point. This will need to be revisited as usage increases but it seems unlikely that this will ever become the system of record for log storage.
Implement Redis based transit of structured json log messages from MW

See also