Fundraising/RFC/Abstract CRM import
This is a proposed extension to CiviCRM, which will facilitate decoupled payment processing and event registration, nightly auditing of donations, and importing from spreadsheet.
The basic framework is that it normalizes inputs and performs various configurable tests on the data quality. New or updated contacts, participations, contributions, and related entities are encoded by a source-specific plugin and then imported in a generalized way.
TODO:
- reconcile with existing work
- plan multi-entity import -- importing a contact and a contribution at the same time is not a standard part of the Civi workflow.
Intake modules
Messaging
A one-directional message-passing protocol in which messages containing an aggregation of entities are sent to CiviCRM by third-parties.
Only a status is returned: was the import successful? We will not pass counts and rejected subentities back across the component boundary—see the transactional model below for how to interpret a success or failure status.
CSV
Auditing
Exception handling
The default will be to wrap each item in a transaction. If some of the message fails, all database changes will be rolled back. Likewise, reads will be made against a snapshot of the database at the beginning of intake.
Metadata
The import processes will stamp various diagnostic and contextual values into the message.
- Source metadata
- source channel, archived raw message URL
- Intake plugin identity
- timestamp at intake, calculated content-type, source revision
- Import job identity
- CiviCRM source revision, hostname, contact.created_time
Fundraising deployment
Introduction of this work into the donations pipeline can be phased.
- Phase 0: write the abstract import library. Not really a deployment phase, but a prerequisite.
- Phase 1: upgrade to an abstract import-enabled version of CiviCRM, without changing message intake configuration
- Phase 2: write an intake plugin for a single source. This should be considered a port of the existing import mechanism for that source, with strict feature parity and tests to prove it. Slow-start this plugin, hand-feeding it small batches. Can run in parallel with legacy import. Discuss upstreaming strategy using example code.
- Phase 3: continue writing intake plugins until finished.
- Phase 4: deprecate legacy intake. Reevaluate coarse architecture, e.g. Drupal vs WordPress.
Data retention
Raw intake data blobs are preserved until there is verification of the imported data's integrity (and possibly longer in long-term storage, for tax or legal reasons?).