Logstash/Extended Retention

From Wikitech
Jump to navigation Jump to search

Policy

See Privacy Policy and Data Retention Guidelines.

Request Process

  1. Ensure the log stream you want to preserve has only non-personal information not associated with a user account.
  2. Create a request to increase retention for a log stream on Phabricator, tagging the Observability team.
  3. Once the audit is complete and approved, the log stream will be tagged for inclusion into long-term retention indexes according to cluster capacity.

Audit Process

  1. Audit the log stream to determine it is non-personal information not associated with a user account.
  2. If in doubt, consult with the Security and/or Legal teams. Notify these teams when necessary.

Common Logging Schema fields indicating PII

These fields (as of ECS 1.7.0) have been identified as likely containing personal information.

  • client.*
  • error.message
  • geo.*
  • http.request.body.content
  • http.response.body.content
  • http.headers.*
  • labels
  • log.original
  • message
  • source.*
  • tls.client.*
  • user.*
  • user_agent.*