Fundraising/Normalized donation messages

From Wikitech

This page documents the in-flight format for donation messages.

Overview

Incoming donations are sent between subsystems as a single packet of data, in a flat dictionary. We use JSON over the wire, and PHP arrays internally. Subscription creations and modifications, refunds, and unsubscribe requests are also sent using similar techniques, and are documented here as well.

We use donation queues to decouple the CRM database from public payments servers, and to not block performance-sensitive tasks.

Legacy format notice: This is the current schema for our messages. Please do not make edits here unless they reflect the existing queue consumer code. Message formats have developed over time and suffer from inconsistency and accumulated cruft, there is a message roadmap with plans to fix this.

Headers

Some data is sent in the message headers in wire format, to make it available for querying.

correlation-id
Certifiably unique identifier sometimes used as the primary key for queuing and dequeuing operations. The default way to construct this ID is GATEWAY-GATEWAY_TXN_ID. If the message does not have a correlation-id, then the native queue server message ID is used instead.
source_name
freeform name of the application which generated this message
source_type
category of application.
payments
came in direct from donatewiki or a banner (payments.wikimedia.org)
listener
real time messages coming in via a listener service created by the payment provider
audit
created by the nightly reconciliation job
direct
message was generated within our CRM system itself, for example a hand-entered donation or refund.
source_host
machine name where initial intake interfacing occurred.
source_run_id
process identifier of the generating code
source_version
revision level of the originating code
source_enqueued_time
unix timestamp encoding when the message was first added to a queue. This header is not updated during requeueing or other operations.

One-time donations

This is the most common type of donation message. They land in the "donations" queue, and are processed by the queue2civicrm consumer.

gateway
Gateway identifier string, e.g. "paypal" or "globalcollect".
gateway_txn_id
Transaction ID string used by the gateway. Sadly, several of our processors cannot guarantee that this is a unique identifier.
date
Time donation was received, in unix timestamp seconds since epoch.
currency
(required) Original currency of transaction. Do not use the deprecated original_currency or original_gross fields, these are too confusing. We'll introduce "settled_currency", etc., when it becomes necessary to track FOREX across processor accounts.
gross
(required) Total amount of transaction, in original currency.
fee
(optional) Fees charged by the payment processor, in original currency. If unspecified, this will be assumed zero, or calculated from gross - net if available.
net
(optional) Amount after subtracting fees.
settled_date
(NOT IMPLEMENTED YET) Unix timestamp at which the transaction was settled.
settled_currency
(NOT IMPLEMENTED YET) Currency code in which settlement was made.
settled_gross
(NOT IMPLEMENTED YET) Total amount settled. (Should this be settled_amount? - it's usually reported after fees)
settled_fee
(NOT IMPLEMENTED YET) Processor fee, in settlement currency.
email
Donor's email. If unspecified, we may substitute with "nobody@wikimedia.org" for validation purposes. This default is stripped out again before storing to the database.
first_name
middle_name
last_name
organization_name
(optional) If given in place of first/last_name, an Organization contact will be created rather than an Individual.
street_address
supplemental_address_1
supplemental_address_2
city
state_province
postal_code
country
Billing or mailing address country—not necessarily the same as the contribution_tracking country of web origin.
payment_method
Primary payment method, e.g. "cc". TODO: enumerate.
payment_submethod
Payment method details, e.g. "visa". TODO
gateway_status
Raw gateway status code, if available.
notes
Text blob will be stored as a CiviCRM note, associated with the contact.
contact_source
TODO WAT?
gateway_account
Our name for the gateway account, e.g. "default", "Euro". We support multiple accounts per processor.
letter_code
TODO
gift_source
TODO
restrictions
TODO
import_batch_number
Conceptually broken, this should be renamed. It's a number internal to AZ Lockbox.
check_number
contact_type
(optional) "Organization" if you want to create an Org record in Civi. This is normally implicit, according to which name fields were filled out.
anonymous
(unused)
optout
(unused)
language
Our best guess at the donor's preferred contact language.
recurring
Always "0" TODO: EH?
utm_campaign
Mapped into the direct_mail_appeal custom field
direct_mail_appeal
contribution_tracking_id
completion_message_id
If a message does not contain enough information to import a donation into Civi, this property gives the ID of a message in the pending queue with the rest of the data. Currently used for AstroPay and Amazon IPN messages.
log_id
Like completion_message_id, but for looking up missing information in payment cluster logs. Used for AstroPay and Amazon audit file messages.

Recurring PayPal donations

gateway
txn_type
PayPal-style recurring subtype. One of (subscr_signup, subscr_modify, subscr_payment, subscr_cancel, subscr_eot, subscr_failed)
subscr_id
Gateway subscription ID
subscr_date
payment_date
mc_currency
mc_amount3
"gross" for all subtypes except subscr_payment
amount3
another name for mc_amount3
mc_gross
"gross" field for subscr_payment
mc_fee
period3
Must be "1 M", meaning we charge every month.
recur_times
Number of installments. "0" means forever (default)
payer_email
first_name
last_name
address_street
address_city
address_zip
address_state
address_country_code
address_name
recurring
Always "1"
cancel
Cancellation timestamp
option_selection1
Premium size
option_selection2
Premium language

Recurring other gateway donations

txn_type
subscr_id
subscr_date
frequency_unit
one of (day, week, month, year), or empty
frequency_interval
installments
recurring
must be 1-ish
cancel

Other fields are the same as for one-time donation messages

Refunds

gateway
type
Refund subtype, one of (chargeback, refund)
date
gross_currency
gross
total charged to us
net
total refunded to donor (not yet implemented)
fee
fee associated with refund or chargeback (may be high for chargebacks, not yet implemented)
gateway_parent_id
Gateway transaction ID of the original transaction
gateway_refund_id
Gateway transaction ID of the refund

Unsubscribe

email
contribution-id

Damaged

When invalid messages are kicked out of a queue, they are copied into the "ORIG_QUEUE-damaged" queue using the following format:

(header) error
WmfException error code, like "INVALID_MESSAGE"
(header) correlation-id
Same as the original message's correlation ID
error
error message
original
json-encoded original message