Fundraising/Development tools/Logdog

Logdog

Logdog is a command-line tool for searching through logs on civi1002 and frlog1002 servers. It supports both content searches within log files and filename-based searches across log directories. It was added to make it easier find transaction data and debug issues in our web logs. Hopefully, in the future, it will be superseded by a more robust solution.

            .--.             .---.
           /:.  '.         .' ..  '._.---.
          /:::-.  \.-"""-;` .-:::.     .::\
         /::'|  `\/  _ _  \'   `\:'   ::::|
     __.'    |   /  (o|o)  \     `'.   ':/
    /    .:. /   |   ___   |        '---'
   |    ::::'   /:  (._.) .:\
   \    .='    |:'        :::|
    `""`       \     .-.   ':/
                '---`|I|`---'
                     '-'

Source

The source code is available on GitHub: logdog.sh

Syntax

logdog [options] query
logdog -f filename
logdog -f filename [filtered_query]

Options

Option	Description
-d YYYYMMDD	Search logs for a specific date (defaults to yesterday for archives)
-o folder_name	Write results to a custom folder name (defaults to query as folder name)
-i	Display the directories that will be searched on the current host
-f, --file	Search for filenames instead of file contents
-h	Display help

Flags

Flag	Description
--first	Limit results to the first match per file scanned

Output

jgleeson@frlog1002:~$ logdog 244536396.2
 ##################################################################
# Searching in /var/log/remote for files matching '*' containing '244536396.2'
# Scanning files: 4% (2/50)
# /var/log/remote/payments-misc hits: 7
# Results written to /home/jgleeson/logdog/244536396.2/payments-misc.txt
Jan 12 10:39:09 payments1006 SmashPig-Gravy: gravy::244536396:244536396.1  | (RawData) New Checkout Session request [] |  | 
Jan 12 10:39:10 payments1006 SmashPig-Gravy: gravy::244536396:244536396.1  | (RawData) newCheckoutSession response: {"cart_items":null,"metadata":null,"buyer":null,"airline":null,"amount":null,"currency":null,"payment_service_id":null,"type":"checkout-session","id":"88a143fa-16ff-43ef-9440-7b02f27231c4","expires_at":"2026-01-12T11:39:10.014259+00:00","payment_method":null} |  | 
Jan 12 10:43:00 payments1006 SmashPig-Gravy: gravy::244536396:244536396.1  | (RawData) Create payment request params: {"amount":200,"currency":"GBP","country":"GB","payment_method":{"method":"checkout-session","redirect_url":"https:\/\/payments.wikimedia.org\/index.php?title=Special:GravyGatewayResult&order_id=244536396.1&wmf_token=4507b19f0bd27641573de3293c9a0dbd%2B%5C&amount=2.00&currency=GBP&payment_method=cc&payment_submethod=&wmf_source=fr-redir.default%7Edefault%7Edefault%7Edefault%7Econtrol.cc&wmf_campaign=spontaneous&wmf_medium=spontaneous","id":"88a143fa-16ff-43ef-9440-7b02f27231c4"},"external_identifier":"244536396.1","statement_descriptor":{"description":"Wikimedia Foundation"},"buyer":{"external_identifier":"jgleeson@wikimedia.org","billing_details":{"first_name":"Jack","email_address":"jgleeson@wikimedia.org","phone_number":null,"address":{"city":"Liverpool","country":"GB","postal_code":"L20 000","state":null,"line1":"123 my street","line2":null,"organization":null},"last_name":"Gleeson"}},"store":true,"payment_source":"card_on_file","user_ip":"123.456.789.123","browser_info":{"user_agent":"Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/143.0.0.0 Safari\/537.36","accept_header":"application\/json, text\/javascript, *\/*; q=0.01","language":"en-gb","color_depth":"24","screen_height":"1080","screen_width":"1920","time_zone_offset":"0","javascript_enabled":true,"user_device":"desktop","java_enabled":false},"connection_options":{"adyen-card":{"window_origin":"https:\/\/payments.wikimedia.org"}}} |  |

Results are displayed and also written to ~/logdog/ in a subfolder named after the query (or custom folder if -o is specified) for follow-up if needed. Each matching file produces a .txt file containing the grep results. e.g. ~/logdog/244536396.2/payments-misc.txt

Search Paths

civi1002

• /srv/org.wikimedia.civicrm/private/log - CiviCRM logs
• /var/log/process-control - Current process control logs
• /srv/archive/civi1002/process-control/ - Archived process control logs
• /srv/archive/frlog/logs - Archived frlog logs

frlog1002

• /var/log/remote - Current remote logs
• /srv/archive/frlog1002/logs - Archived frlog logs
• /srv/archive/civi/process-control/ - Archived civi process control logs

Examples

Search for an order ID in logs:

logdog order_id_12345

Search for files matching a pattern:

logdog -f payments-paypal

Search within matched files for specific content:

logdog -f payments-paypal-202411 paypal_order_id_sometime_in_november

Search logs from a specific date:

logdog -d 20241115 order_id_12345

Get only the first match per file:

logdog --first order_id_12345

View which directories will be searched:

logdog -i

Notes

• Logdog automatically selects the appropriate grep tool (grep, zgrep, or bzgrep) based on file extension
• Filename searches are limited to 50 results by default
• Partial dates (less than 8 characters) are treated as wildcards
• The script only runs on civi1002 or frlog1002; running on other hosts will produce an error