Logstash/Extended Retention

Policy

See Privacy Policy and Data Retention Guidelines.

Request Process

Ensure the log stream you want to preserve has only non-personal information not associated with a user account.
Create a request to increase retention for a log stream on Phabricator, tagging the Observability team.
Once the audit is complete and approved, the log stream will be tagged for inclusion into long-term retention indexes according to cluster capacity.

Audit Process

Audit the log stream to determine it is non-personal information not associated with a user account.
If in doubt, consult with the Security and/or Legal teams. Notify these teams when necessary.

Common Logging Schema fields indicating PII

These fields (as of ECS 1.7.0) have been identified as likely containing personal information.

On the labels, message, and log.original fields: The code paths populating these fields will need to be audited and demonstrate that personal and/or nonpublic information cannot be inadvertently written to these fields.

client.*
error.message
geo.*
http.request.body.content
http.response.body.content
http.headers.*
labels
log.original
message
source.*
tls.client.*
user.*
user_agent.*