mwlog
mwlog are a class of logging hosts which serve multiple functions.
Hosts
Services
udp2log/mwlog
Ingest and storage of "udp2log" logs, in flat file format in /srv/mw-log.
These are logs transported over udp/8420 to an rsyslog listener on the mwlog hosts which mirrors the logs to the udp2log listeners on all mwlog hosts (udp/8421)
udp2log/mwlog producers are configured in various different ways:
- Mediawiki uses hard coded IPv4 addresses (intentionally, for performance reasons) for the mwlog destinations, configured in mediawiki-config/wmf-config/ProductionServices.php
- Some services (including rsyslog which relays on behalf of php-fpm and apache2 ) use the udplog.(eqiad|codfw).wmnet DNS records. Updating the udplog DNS record will require an rsyslog restart across the fleet.
- Some services are defined via puppet hiera (can be found with e.g. git grep mwlog1002 in operations/puppet)
Redis server for Arc Lamp
The primary (eqiad) mwlog host recieves sampling profiles from MediaWiki app servers (collected by Excimer), which are then subscribed to by Arc Lamp (formerly known as "webperf" or "xenon").
Long-term security storage
Currently /srv/security on the mwlog primary (eqiad) is used as a long-term store for security log archives.
Utilities
logspam-watchdisplays and filters a summary of recent logs from disk - see Wikimedia binaries#mwlog-server