Fundraising/Development tools/Logdog

Logdog

Logdog is a command-line tool for searching through logs on civi1002 and frlog1002 servers. It supports both content searches within log files and filename-based searches across log directories. We added it to make it easier to find transaction data and debug issues in our web logs. Hopefully, a more robust solution will supersede it in the future.

            .--.             .---.
           /:.  '.         .' ..  '._.---.
          /:::-.  \.-"""-;` .-:::.     .::\
         /::'|  `\/  _ _  \'   `\:'   ::::|
     __.'    |   /  (o|o)  \     `'.   ':/
    /    .:. /   |   ___   |        '---'
   |    ::::'   /:  (._.) .:\
   \    .='    |:'        :::|
    `""`       \     .-.   ':/
                '---`|I|`---'
                     '-'

Source

The source code is available on GitHub: logdog.sh

Syntax

logdog [options] query
logdog -f filename
logdog -f filename [filtered_query]

Options

Option	Description
`-d YYYYMMDD`	Search archives from this date. A prefix like `202610` matches the whole month. Without `-d`, current logs and the latest archives are searched
`-o folder_name`	Write results to a custom folder name (defaults to query as folder name)
`-i`	Display the directories that will be searched on the current host
`-f`, `--file`	Search for filenames instead of file contents
`-h`	Display help

Flags

Flag	Description
`--first`	Limit results to the first match per file scanned

Output

Results are written to ~/logdog/ in a subfolder named after the query (or custom folder if -o is specified). Each matching file produces a .txt file containing the grep results.

Search Paths

civi1002

/srv/org.wikimedia.civicrm/private/log - CiviCRM logs
/var/log/syslog - Current syslog
/var/log/process-control - Current process control logs
/srv/archive/civi1002/process-control/ - Archived process control logs
/srv/archive/frlog/logs - Archived frlog logs

frlog1002

/var/log/remote - Current remote logs
/srv/archive/frlog1002/logs - Archived frlog logs
/srv/archive/civi/process-control/ - Archived civi process control logs

Examples

Queries are passed straight to grep -i, so they're case-insensitive and treated as regular expressions. Quote anything with spaces or special characters.

Everyday searches

Find everything about an order ID in the current logs and latest archives:

logdog 12345.1

Search for an email address. Escape the dot, or it matches any character:

logdog 'donor@example\.org'

Search for a phrase. Spaces become underscores in the output folder name (~/logdog/payment_failed/):

logdog 'payment failed'

Match several IDs at once with a regex alternation:

logdog '12345\.1\|12346\.1\|12347\.1'

Busy queries

Only check whether each file mentions something, without pulling every line. Useful for common terms that would otherwise produce huge output:

logdog coworker --first

Put the results of a longer investigation in a named folder, so later searches don't mix with them:

logdog -o T123456-refunds 'refund.*12345\.1'
ls ~/logdog/T123456-refunds/

If one path is taking too long (e.g. /var/log/process-control has thousands of files), press Ctrl-C once to skip it. The remaining paths are still searched, and the summary at the end shows which paths were skipped and how far they got.

Older logs

Search the archives for a specific day:

logdog -d 20261005 12345.1

Search a whole month. A partial date matches everything that starts with it:

logdog -d 202610 12345.1

Search a range of days with a glob:

logdog -d '2026101[0-2]' 12345.1

Filename searches

List log files whose name contains a string (alphabetical, up to 50 per path):

logdog -f payments-paypal

Narrow to one month's PayPal files, then search inside only those files:

logdog -f payments-paypal-202610 12345.1

Before you start

Check which directories will be searched on this host:

logdog -i

Notes

Logdog automatically selects the appropriate grep tool (grep, zgrep, or bzgrep) based on file extension
Press Ctrl-C to skip the path currently being searched and move on to the next one. Press it twice within 2 seconds to quit. A summary of hits per path, including skipped paths, is printed at the end
--first and --file can go anywhere in the command, e.g. logdog order_id_12345 --first. Short options (-d, -o, -i, -f) must come before the query
Filename searches are limited to 50 results by default
Partial dates (less than 8 characters) are treated as wildcards
The script only runs on civi1002 or frlog1002; running on other hosts will produce an error