Exim
Troubleshooting "exim queue warning" alerts
Metrics dashboard: https://grafana.wikimedia.org/d/000000451/mail
This alert fires when the number of queued emails on an Exim server exceed the defined threshold. Usually about 1000 messages. A few possible causes are:
- A remote domain who receives a lot of our email is down, and we are queueing messages for redelivery.
- A user who receives a lot of our email has a problem with their mail inbox (over quota, account removed, etc.) and we are queueing messages for redelivery.
- Problematic messages are being relayed through our mail system and are being temporarily rejected, causing us to queue valid mail for redelivery (rate limiting, spam prevention, etc.)
To get a better understanding of why this alert happened you can try reviewing the size of the queue on the relevant host(s) and look to see if the problem is localized to a single domain, or if it affects multiple domains. For example:
root@mx1001:~# mailq | exiqsumm -c
Count Volume Oldest Newest Domain
----- ------ ------ ------ ------
881 5700KB 4d 0m gmail.com
21 104KB 126d 126d wikipedia.org
16 47KB 259d 11d mediawiki.com
14 35KB 259d 39m wikibooks.com
13 38KB 259d 70d wikipedia.in
11 23KB 259d 18d wiktionary.com
9 21KB 259d 13d wikiversity.com
In this case the vast majority of mail in the queue is for gmail.com. Good, now we know the problem is not affecting multiple domains. Now we might wonder how many users at that domain, in this case gmail.com, are affected. We can get a quick count of deferred messages for a given domain with something like the following
mx1001:~# mailq | grep gmail.com | grep -v D | sort | uniq -c | sort -n
2 foobar@gmail.com
15 darthvader@gmail.com
40 bazbaz@gmail.com
812 redacted@gmail.com
Ok, from this we can see user redacted@gmail.com is responsible for the vast majority of queued messages. With this info we can then look for a reason in the exim logs. This may take some hunting and pecking through the logs depending on the nature/scope of the problem. Something like the below is a good start and will often give hints
# grep -i $problem_domain /var/log/exim4/mainlog | grep -i error
# for example:
mx1001:~# grep -i redacted@gmail.com /var/log/exim4/mainlog | grep -i error
# example error:
H=alt2.gmail-smtp-in.l.google.com [209.85.202.27]: SMTP error from remote mail server after RCPT TO:<redacted@gmail.com>: 452-4.2.2 The email account that you tried to reach is over quota. Please direct\n452-4.2.2 the recipient to\n452 4.2.2 https://support.google.com/mail/?p=OverQuotaTemp s27si1505313edm.307 - gsmtp
If errors are present you should see an indication of if they are 4xx (temporary) errors or 5xx (permanent) errors, and a short description of the problem as provided by the recipients mail system.
In the example above we can see a recipient is over quota, and their mail provider is temporarily rejecting messages with a 452 code, so our mail server is queueing them. If the user receives a lot of mail this can push the check over the alert threshold.
show mail queue
mailq exim -bp
flush mail queue
runq exim -q
force delivery attempt
exim -qf (non-frozen messages) exim -qff (all messages, frozen or not)
deliver just one specific mail from queue
exim -M [queue-id]
(the queue-id is what you see after the size and before the email address)
search for specific mails in the queue
exiqgrep -f <sender address> exiqgrep -r <recipient address> man eqixgrep for more options
count number of mails to a specific recipient
exiqgrep -cr <recipient address>
remove emails to a specific recipient
exiqgrep -i -r specific@recipient.org | xargs exim -Mrm
test address routing
On e.g. mx1001.wikimedia.org
exim -bt <address>