Fundraising/Data and flow/Audits
Overview
This documents the workflow to process audit files from payment processors and import missing messages into CiviCRM.
For all payment processors, except PayPal, the WMFAudit.parse api call processes the audit files. The code in the civicrm extension handles reading the list of files from the directory, searching for existing transactions in the database, and finding missing information in the payments-wiki logs (mounted at /srv/archive/frlog/logs) for each transaction that isn't in the database. The code to parse the individual files to an array of normalized transactions lives under the SmashPig codebase, in classes that implement the AuditParser interface.
Audit files are located on civi1001 in /var/spool/audit/[payment-processor] and divided into two directories: incoming and completed.
How to run locally
Run the audit parser for one gateway:
cv api4 --user=admin -vv WMFAudit.parse gateway=adyen logSearchPastDays=12
Run with just one specific file:
cv api4 --user=admin -vv WMFAudit.parse gateway=adyen logSearchPastDays=12 file=payments_accounting_report_2024_04_06.csv
Refunds
If a refund is successful it will put a message on the refund queue:
cv api4 --user=admin -vv WMFQueue.Consume timeLimit=280 queueConsumer=Refund queueName=refund
Schedule and Files
Adyen
Runs every night at 20:17 UTC
payments_account_report - nightly file, has the previous days transactions
settlement_detail_report_batch - weekly file every Friday, has the previous weeks transactions
Amazon
Runs every night at 00:23 UTC
settlement_data - weekly file on Tuesday
refund_data - only if there are refunds, not always present, weekly file on Tuesday
Braintree
Runs every night at 00:00 UTC
settlement_batch_report - nightly file, has previous days transactions
settlement_batch_report_dispute - nightly file, has previous days disputes
settlement_batch_report_refund - nightly file, has previous days refunds
Dlocal
Runs every night at 00:20 UTC
files - nightly
Gravy
Runs every night at 21:17 UTC
files - nightly
Ingenico
Runs at 01:00 UTC on every day-of-week from Tuesday through Sunday.
wikimedia_report - nightly file, has previous days transactions
Paypal
Runs nightly at 17:35 UTC
TRR - nightly (what is this
STL - nightly (what is this)
WIkimedia - nightly file, has previous days transactions
Fundraiseup
Runs nightly at 01:00 UTC
Including new donations, new recurring, recurring cancellations, and refunds.
Payment Processor Specific Information
Adyen
Fundraising/Data and flow/PSP integrations/Adyen Checkout#Audits
Amazon
Instead of an SFTP download, we have to call methods on the Amazon Pay SDK to get our reports. This is kicked off with the DownloadReports php script in SmashPig.
Dlocal (formerly Astropay)
Fundraising/Data and flow/PSP integrations/dLocal#Audits
Braintree
Using graphQL run and consume the report nightly
Fundraiseup
Process control jobs
fundraise-up_audit.yaml - Runs at 1AM UTC daily. Calls fundraise-up_audit_download and then fundraise-up_audit_parse.
fundraise-up_audit_download - Downloads fundraiseup export files from the Fundraiseup SFTP server. This files contains the new donations, new recurrings, cancelled recurring, and refunded transactions from Fundraiseup.
fundraise-up_audit_parse - Imports the transactions from the exported files. Calls cv api4 --user=admin -vv WMFAudit.parse gateway=fundraiseup logSearchPastDays=12
PayPal
PayPal uses a python script deployed in the fundraising tools repository: https://phabricator.wikimedia.org/diffusion/WFTO/browse/master/audit/paypal/
We currently use PayPal's Express Checkout integration, and classify those donations using gateway value 'paypal_ec'. However, we still have some legacy recurring donations from the old integration, and those are classified with gateway value 'paypal'.
Log
check audit parse log from frlog1002, located at /srv/archive/civi/process-control/<yyyymmdd>/<paymentMethod>_audit_parse-<yyyymmdd>-xxxxxx.log.civi1001.bz2
Result print example as:
Done! Final stats:
Total number of donations in audit file: xxx
Number missing from database: xxx
Missing transactions found in logs: xxx
Missing transactions not found in logs: xxx
Missing transaction summary:
Regular donations: 2
Returned from hook drush_wmf_audit_parse_audit [1.13 sec, 36.01 MB] [debug]
xxxxxxxx: 1 xxxxxxxx: 1
Refunds and chargebacks: 0
Recurring donations: 0
Command dispatch complete [1.13 sec, 35.92 MB] [notice]
Transaction IDs:
xxx xxxxxxxxxx xxx xxxxxxxxxx
Initial stats on recon files: Array
(
[/var/spool/audit/xxx/incoming/xxx] => 0 [/var/spool/audit/xxx/incoming/xxx] => 0 [/var/spool/audit/xxx/incoming/xxx] => 2
)
File Wrangling
Sometimes the audit processor can't resolve all the transactions in a file, even after trying for several days. This can lead to a build-up of files in the incoming directory and to subsequent processor runs getting longer and longer till finally they start timing out. The solution is to 'manually' move the older files from the incoming to the completed directory. Since our personal accounts don't have permissions to move the files, we do this with a one-off process-control job such as ingenico_move_audit_files. Since process control runs each command as a separate process under python, we need to wrap any file globs that we want expanded with 'sh -c', for example:
sh -c "mv /srv/archive/civi1001/audit/globalcollect/incoming//wx1*202010[01][0-5]*xml* /var/spool/audit/globalcollect/"
Adding a New Payment Processor
Once the new processor code has been added
1. Enable the Module
2. Add the folders
This is done via puppet by adding the processor name to the $audit_processors array in modules/civicrm/manifests/audit.pp. Additionally, some files (YAML usually) may be needed for the audit configuration. Those vary by processor but are stored in the same audit.pp manifest.