mwlog
mwlog are a class of logging hosts which serve multiple functions.
Hosts
Services
udp2log/udp_tee (MediaWiki/Scap logging)
Ingest and storage of "udp2log" logs, in flat file format in /srv/mw-log.
On all mwlog hosts, rsyslog listens on 8420/udp to provide the udp_tee functionality (profile::rsyslog::udp_tee). The udp_tee listener receives logs from udp2log producers, such as MediaWiki and Scap.
udp_tee mirrors the received UDP logs to udp2log instances on all mwlog hosts, listening on 8421/udp (code). Even though udp2log producers to a single mwlog host running udp_tee, this allows all udp2log instances to process 100% of the logs emitted by the udp2log producers. Theoretically, this allows for a hot standby scenario where failover does not result in loss of logs sent *after* enabling mirroring.
udp2log/mwlog producers are configured in various different ways:
- MediaWiki emits to a single
udp_teeinstance. See: MediaWiki UDP logging#udp2log. - Some services (including rsyslog which relays on behalf of php-fpm and apache2) use the
udplog.(eqiad|codfw).wmnetDNS records. Updating the udplog DNS record will require an rsyslog restart across the fleet. - Some services are defined via puppet hiera (can be found with e.g.
git grep mwlog100in operations/puppet)
As said: the udp2log binary, running on all hosts, receives logs from udp_tee. udp2log pipes messages to executables defined in /etc/udp2log. In most cases, this is demux.py. demux.py writes each message to a per-channel log file in /srv/mw-log See: Logs#mwlog:/srv/mw-log/.
Due to privacy reasons, logs in /srv/mw-log are subject to a retention policy.
Long-term security storage
Currently /srv/security on the mwlog primary (eqiad) is used as a long-term store for security log archives.
Former services
Utilities
logspam-watchdisplays and filters a summary of recent logs from disk - see Wikimedia binaries#mwlog-server