Fundraising/Internal-facing/CiviCRM/ContactMatching

Contact Matching Logic

When we process donations from donors giving online on donatewiki, we use a number of strategies to match donors to existing contacts in CiviCRM. Full details on this process are provided below, but on a high-level the process works by matching by name and email.

Because we do not collect all of the donor information directly on the donation form when donor gives by ApplePay, GooglePay, Venmo, Paypal or ACH, we may get an email address that the donor uses specifically for that service and we may get a name that differs from what they might have provided in the past. To make it feasible to match contacts in these cases, we consider names and emails from these sources as low confidence. We store these emails with location types of the same name in CiviCRM (e.g. Apple, Google, etc), which allows a donor to have different primary email address from the email they use for one of these services, while still allowing for matching. This can cause some donors to have more than one email of the same email on their contact record, but this is a worthwhile tradeoff for improved matching and preventing a contact's primary email, which is what we use to communicate with them, from being changed.

When a contact has a primary email which is one of the above types, we know not only that we have lower confidence in the email, but also that we have lower confidence in their name, which we received from the service.

For ACH specifically, the donor provides an email on the form and the bank may provide an email via Trustly, which may be different. We store the donor provided email as a ACH-form email, which we have high confidence in, and the bank provided email as an ACH-bank email, which we have lower confidence in.

The downside of this change may be that some donors will receive their thank you email on their primary email instead of the email they used to donate via one of these services. In some cases this may be good (that's their main email, they don't check the email attached to Venmo) and in other cases bad (they didn't get their receipt because it went to an older email). If this becomes an issue for DR, we can refine further.

Here's a simplified overview of the matching process:

• Match on contact id if provided (e.g. the donor gave from a personalized link in an email from Acoustic)
• We will also match on Venmo username or phone number from Venmo, if these are available
• We will always first match on name and email if possible
• If we have high confidence in the incoming data, i.e. it was provided on the form by the donor, then we match to a contact with the same email who has a name we have low confidence in (and we'll change their name to the new one).
• If we have low confidence in the incoming data, we match on email only, prioritizing primary emails and emails with the same type as the incoming (but we won't update their name).
• We'll try to match by the contact's address (requiring street address, city and postal).
• If we can't find a match through any of these, we create a new contact.

If Donor Relations are editing a contact and they understand that an email the donor's primary email has changed, they can change the email as usual. If that email is currently a type like Apple, Google, etc, it would be best to change the type to Home so that we know that we have high confidence in this email — but nothing terrible will happen is this change isn't made.

Updating email addresses for Paypal, etc donors

With the above, we can now update email addresses for Paypal donors. All we need to do is to change the current primary email address (the email that is attached to the Paypal account) to a Paypal type and then we can safely add a new email address of Home type and make the new email primary. Then we will match incoming donations to the contact based on the Paypal type email address, but we won't update the primary email address.

The same can be done for donors who give with ApplePay, GooglePay, Venmo or ACH, selecting the appropriate email type (using ACH-bank for ACH donations).

Full Contact Save Logic

Contact Matching UML

UML

link: https://gitlab.wikimedia.org/repos/fundraising-tech/fr-tech-diagrams/-/tree/main/contact_save

Contact matching logic -> Find Existing Contact

The matching logic is centralized in the getExistingContact() method. It attempts to find a record using a specific hierarchy of strategies. If multiple matches are found, it always returns the oldest contact (the one with the lowest ID).

Matching Strategy Order:

1. Direct ID:

If a contact_id is provided in the message, fetch that exact contact immediately. If the contact has been merged to another contact (and not yet fully deleted, which happens after a year), then get the contact that it has been merged to.

2. Name Validation (Stop Condition):

• If names (first_name/last_name) are missing and the source is not a low-confidence source (like Apple Pay), the search stops and returns NULL.
• If names are missing but the source is low-confidence, skip name-based matching to avoid false positives.

3. External Identifiers:

Look up by custom fields (e.g., Venmo username, PayPal ID).

• Special Case (Venmo): If paying via Venmo, check Phone or Venmo Email first to avoid false matches (since Venmo usernames change), falling back to the custom field only if those aren't found.

4. ACH Billing Email:

Check if a contact exists where the email matches the provided billing_email tagged with location type 'ach'.

5. General Email Match:

Match by email with different confidence level (priority order):

• 1) Primary Email + Name match.
• 2) Email + Name match.
• 3) Primary Email with a Low Confidence Source type (e.g., venmo, apple).
• 4) Email matching a specific low-confidence location type (e.g., 'venmo').
• 5) Email with a Low Confidence Source type (e.g., venmo, apple).

6. Address Fallback:

If nothing else matches, it searches by Address (street_address, city, postal_code, plus names) provided the addresses are long enough (min 5 chars for street, 2 for city/zip).

Contact Save UML

How to Assign Email (On Create)

If getExistingContact returns NULL (no match found), the system proceeds to Create:

• Primary Email: Set via email_primary.email.
• Location Type Assignment:
  • If a payment_method is provided (e.g., Apple Pay, Venmo), the email location type is set to that specific method (e.g., applePay, venmo) to mark it as a secondary/external source.
  • If the payment method is ACH, the location type is set to 'achForm'.
• Billing Email: If billing_email is provided separately, it is saved as a secondary email with location type 'ach'.
• Opt-In/Out: Fields like Communication.opt_in and do_not_solicit are handled via specific API fields.
• Phone Consents: If an SMS phone number is present and opted-in, a PhoneConsent record is created.

How to Update Email (On Update)

When an existing contact is found, handleUpdate() is called, which specifically triggers emailUpdate(). This logic determines whether to overwrite, update a secondary email, or create a new secondary email.

Step A: Determine Source Trust

• Trusted Sources: Anything not listed as a 3rd party (e.g., standard website forms).
• Untrusted Sources (Low Confidence): google, apple, venmo, paypal, and achForm.
  • Exception: Even if billing_email is provided for ACH, the primary email logic still treats the form email as trusted if the payment method isn't ACH.

Step B: The Update Decision Tree (updateEmailBasedOnContext) The system checks three conditions in order:

1. Exact Match Primary:
   • If the incoming email matches the current primary email:
     • If the incoming source is Trusted and the current primary is Untrusted: Update the primary to the trusted source.
     • If the incoming source is Untrusted (e.g., Apple Pay) and the current primary is Trusted: Keep current primary.
     • If the incoming source is ACH (achForm) and payment is NOT ACH: Update location type to Home (Trusted).
2. Location Match:
   • If the email matches an existing email with the same location_type_id (e.g., existing venmo email):
     • If the address is the same: Do nothing.
     • If the address changes: Update that specific location's email. Mark it Primary if the source is Trusted/Current primary is untrusted; otherwise, keep it Secondary.
3. Create Secondary:
   • If no match is found for the incoming email:
     • If the source is Trusted: Make the new email the Primary.
     • If the source is Untrusted: Create a new Secondary email with the corresponding location type (e.g., venmo, google).

Step C: Special Handling for ACH Billing There is a separate check for handleBillingEmailIfPresent.

• It ensures that even if the Primary email matches perfectly and is trusted, if there is a separate billing_email provided in the message (and no ACH email exists), the system creates a secondary ach email.
• It updates the existing ach email if the new billing_email value differs.

Summary of Email Logic Rules

• Trusted > Untrusted:

A trusted email (web form) will always become the primary email over an untrusted one (Apple Pay), even if the Untrusted one was previously primary.

• ACH Priority:

ACH emails are treated as secondary billing addresses, regardless of the primary email source.

• No Overwrite on Low Confidence:

If an existing primary email is trusted, we generally don't replace it with a new low-confidence email unless the existing one is invalid or the incoming source is trusted.

• Address Changes:

If an address changes (city/zip) on an existing record, the system updates the address data but relies on the email logic above to decide if the contact record itself should change.