CiviCRM requires a 'host CMS' and to that end we use Drupal7. Drupal 7 is EOL in November 2022 and next year we plan to upgrade to Standalone CiviCRM, Drupal 9 - or maybe even 10. Our goal, however, is that we do not use any CMS-specific code going forwards. While we currently expect to stick with Drupal in we should be equally able to move to Wordpress. To this end we are in the process of migrating our drupal modules to CiviCRM extensions.
Drush is a really useful drupal command line utility. There is a lot of documentation about drush on the internet but a few things to know with regards to WMF.
On production, staging and in our docker dev set up we have an alias 'wmff' which tells drush details about where the code is and to use user 1.
Common usage:
Command
Where
What
`drush @wmff updb`
local dev and prod
Run any database updates that need to be run
`drush @wmff up --security-only`
local dev
Download and install any security updates (these are then checked into git to deploy)
If you see problems finding classes while enabling the civicrm extension, this may be due to stale metadata cached in redis. You can use the queues-redis-clear.sh script to clear that out (note you will also lose all messages in the donation and job queues).
Log Tables
We use CiviCRM logging to create log tables which we use for
- Threshold queries - query log_civicrm_contribution for contributon updates in the last 5 minutes (not includes any sort of update)
- Forensics - eg https://phabricator.wikimedia.org/T311438 involved figuring out history of a contact around 18 months ago. Importantly determined they never had any contributions - ie didn't have them & then they were deleted, but never existing
Or best practices
CiviCRM uses the drupal whitespace standards (sadly adopted before the world co-alesced on something different) so all drupal modules & civicrm extensions (including the CiviCRM codebase) should use that. When using phpstorm installing the drupal plugin will add this standard. Within the vendor directory PSR2 is likely more appropriate
We agreed to name our civicrm extensions with the prefix 'wmf-` where they are WMF specific and just the name if they are to be shared - ie `wmf-civicrm` and `deduper` respectively. This applies going forwards (we may or may not fix existing ones)
Running CiviCRM behind a firewall
In order to run CiviCRM behind a firewall we have disabled some SystemChecks on production (but not on our local developer environments) and use a ckeditor image uploader that embeds images. These are documented in the main CiviCRM firewall documentation. In our case the use of browser certificates for authentication means the server cannot reach it's own urls during crons
We supplement Core deduping behaviour with the Deduper extension - which we also maintain as an open source extension.
We have 4 ways of tackling duplicate contacts in CiviCRM
When piping contacts into CiviCRM we do a basic strict lookup based on first name, last name and email to try to match the contact
When manually importing we implement various rules - see Importing Data
We have an automated script that runs through our most recently modified contacts all the time (based on civicrm_contact.modified_date) and attempts to run dedupe in safe mode. This includes various conflict resolution strategies described in the Deduper extension documentation and configured on our site at civicrm/admin/setting/deduper.
We have started to use the CiviCRM queue mechanism for some of our processes - with a view to using more broadly as we move away from Drupal7.
The main queue documentation is on gitlab but some updates that we have deployed on production have not (as of Nov 2023) been updated to gitlab. Notably this includes new api interactions which allow us to run the queue from drush for debugging.
Setting the queue paused Setting will not interfere with the current queue item being acted on, but once it is released no new items can be claimed.To see what is currently either claimed or stalled try the query
More or less do what coworker does but end when finished rather than keeping monitoring.
Queue log output
If there is an error processing a queue item it will output to syslog - you can grep for 'civicrm.queue' to find it (or civicrm.queue.ERROR to only find errors). Note the firtst part
(Task "omni-snooze#366920") gives both the queue_name and the queue_item id. The portion after the full stop is the error message from the failed action
civicrm.queue.ERROR: Task "omni-snooze#366920" failed and should be retried. Request failed: Recipient must be existing, and OPTED IN or SNOOZED, and (truncated)
Running coworker locally on the docker container, cd into the /srv/civi-sites/wmff/drupal and run the following commands to initialise CiviCRM and initiate the pipe for the process:
coworkerrun-vvv--pipe='drush ev "civicrm_initialize(); Civi::pipe();"'
Viewing the status of a queue
The status of the queue can be found on the civicrm_queue table on the civicrm database. Statuses include: pending, aborted, and active.
Automated emails from CiviCRM
We send out the following automated emails
Recurring failure notifications - these are send when a monthly recurring email is failing and encourages people to set up a new one. It is not sent if they have an active recurring email.
Thank you letters - these are send by an automated job for every donation in CiviCRM, unless the 'no_thankyou' field is populated or it is a recurring donations
End of year emails - these are sent at the start of the year to cover all the recurring contributions in the previous year. These can also be sent ad hoc to individual donors (in which case they include all donations, not just the recurring ones)
Thank you , monthly convert, recurring failure notification and end of year email text & translations
The text for these emails is managed within CiviCRM in the Message Templates interface. using Smarty template language (version 2). This allows for the use of some variables and some conditional logic.
For example most templates have a block like the annotated one below. This code block uses the variables $first_name, $last_name, $contact_type and $email_greeting_display to construct a greeting
{if$first_nameand$last_name}// If we have both namesDear{$first_name},// then use the first name as a salutation {elseif$contact_type==='Organization'&&$email_greeting_display}// Otherwise, if they are an organization with a saved greeting {$email_greeting_display},// use the saved greeting{else}// otherwise Deardonor,// use a standard greeting (appropriate to the translation){/if}// always close the if!
Money formatting in emails
The {$amount} assigned to the template is formatted according to the relevant language - for example in Canadian French it renders to 4 000,99 $ US but in Vietnamese it displays as 4.000,99 US$
One gotcha with this formatting is the library assumes the currency is implicit when the currency matches the language. For example in the English United States template it does not specify the currency when the currency is USD. Fortunately (once https://phabricator.wikimedia.org/T341101 goes out) we can assume that each language should have one currency it might drop the try currency for and we can add it back on a per-template basis.
The table below shows how we can add the iso code for USD only. It also shows that it only makes sense to do this when the language is English (United States) because in other languages it is already being added as it is not implicit, resulting in double ups. So the best option for English (Unitied States) templates is
{$amount} {if $currency ==='USD}{$currency}(/if}
but for the portuguese template it is simply {$amount}.
However, Donor Relations makes use of the ISO codes down the bottom so duplication there appears to be OK
Currency
Language
Template Code
Displays
CAD
English (United States)
{$amount}
$CA 50.00
USD
English (United States)
{$amount}
$50.00
USD
Portuguese (Portugal)
{$amount}
4 000,99 US$
CAD
English (United States)
{$amount} {$currency}
$CA 50.00 CAD
USD
English (United States)
{$amount} {$currency}
$50.00 USD
USD
Portuguese (Portugal)
{$amount} {$currency}
4 000,99 US$ USD
CAD
English (United States)
{$amount} {if $currency ==='USD}{$currency}(/if}
$CA 50.00
USD
English (United States)
{$amount} {if $currency ==='USD}{$currency}(/if}
$50.00 USD
USD
Portuguese (Portugal)
{$amount} {if $currency ==='USD}{$currency}(/if}
4 000,99 US$ USD
CiviMail records
When we send out thank you emails we track them in the CiviMailing system - even though they were not initiated by civimail. This attaches a verp address as explained in the CiviCRM documentation. In order to match any replies or bounces we get from this we need a record in the `civicrm_mailing_event_queue` with the queue_id and hash. This is of time-limited value - once we no longer care about the related mailing event records we don't need the queue record. The query below gives data about the mailing event records.
Currently we only have current data in 3 of the tables
- bounces (We are now creating activities for any received bounces or replies)
- replies (<500 in total)
- delivered events (these are created when we create the mailing records - they really aren't of much value to us.
CiviCRM supports sending out manual emails to small groups but expects you to use CiviMail to send to larger groups. We have configured out site to permit sending to 700 rather than the normal 50 under the 'small group' category via the `simple_mail_limit` setting. This is a hidden setting because there are implications to having a larger group in that
1) sending lots of emails without such things as unsubscribe options and organization details can negatively affect an mail servers reputation, affecting deliverability. In our case this is somewhat mitigated by the fact that this is a small portion of the email we send
2) bounce (and potentially unsubscribe requests) are not handled
Although the 'classic' way to use CiviMail is through the CiviMail interface it can be done from search results - which is the likely usage for WMF staff. In that case it still creates a group within CiviCRM for the receipients but the group is hidden from the user interface.
When CiviMail is used the 'from' address is changed to a 'verp' address - ie a hash followed by a dot and the email address of an internal email that we process via a scheduled job
Custom fields in CiviCRM can be created through the user interface. In order to allow flexibility to our users the arrangement we have with our super-users (Nora, Rosie) is that they can create custom fields through the UI but they should create a phab task so that fr-tech can follow up ( add field to advance search and so on ).
The follow up by frtech is in 2 parts - ensuring the fields are present in our dev environments and creating triggers.
Ensuring the fields are present in our dev environments
Keeping our dev environment fields in sync is a best-efforts endeavor rather than something we keep 100% in sync, but it does make it easier for us to develop locally. There is currently an 'old method' and a 'new method. The new method is to use the CiviCRM managed entities functionalityas reflected in `CustomGroupMatchingGiftInformation.mgd.php` file. The previous barriers to this standardised approach, around the use of the more efficient BulkSave action and matching existing entities, are now resolved. However we DO set the action on update & cleanup to 'never' to avoid anything unexpected on production. The command to reconcile these is
drush@wmffcvapiManaged.reconcileversion=4
The old method involves tracked/synced fields being declared in the CustomGroups.php file in the Managed directory in the `wmf_civicrm` extension. '. This file follows the conventions of the CiviCRM managed entities functionality and the fields declared in the file are added to our developer builds on install.
However, because the file is not a direct match to the Custom fields on prod, we have not registered this file with the civicrm_managed hook, and instead we have a custom wmf command which adds any declared fields in the CustomGroups.php, that are missing for the dev site.
This command adds CustomGroups and CustomFields to dev sites if missing, but does not update them. It only creates option values if the field did not previously exist or it is being run in a development environment - ie we want to add but not update on live.
Note that
When declaring the fields in the CustomGroups array an easy way to get the data from live is to use the API v4 explorer - once the criteria are selected & execute has been hit the field data is listed in a json format and there is even an option next to it to switch to a php format. Fields that do not differ from the defaults (including is_active) should be removed from the resulting array, along with the id field. The explorer can be used in a similar way to get the CustomFields in the group and any option values (using the option_group_id from the custom field definition). Do not include option_group_id or custom_group_id in the checked in array
In CivICRM all field types can be extended with custom groups - however, CiviCRM must know that they can be extended. CiviCRM has a hard coded mapping of the common entities (Contribution, Contact etc) but also maintains an option group ` cg_extends` with other entities. When extending an entity type that is not extendable by default we need to ensure the option value exists.
In some cases the functionality of the custom fields are owned by extensions rather than WMF user driven. In these cases the fields are declared in the relevant extension (e.g the Omnimail extension installs 2 custom groups and the relationship block extension installs one). These are written into the Upgrade classes in the relevant extensions and extension upgrades are run using the following command.
drush@wmffcvapiExtension.upgrade
Update triggers on production
The process for updating the triggers we use for logging doing this is in the subpage
You can run the following drush command on dev or staging to import (60) contributions and get timings.
drush@wmffqperf-d60"your comment"
Query tracking
If you wish to see what queries run you can add this line to your civicrm.settings.php file - the 'n' is just part of the file name, in case you want to do separate runs
define('CIVICRM_DEBUG_LOG_QUERY','n');
You can also capture just the queries for one command using env - eg. this will log the queries for the command in question but not other queries
envCIVICRM_DEBUG_LOG_QUERY=deleteddrush@wmff-vvcivicrm-cleanup-contacts--batch=500--date_cutoff="1 year ago"
The extension org.wikimedia.systemtools provides a script / api to help analyse this file - you may need to installl it - in the UI it is called "Home for WMF helpers".
Once enabled you can run the following
It will output a csv with a cleaned up version of the queries to the directory you are in - the output will give the filename (spoiler - it's gonna be query_log_parsed.csv).
Also note the extension has a Readme.md....
At this point I generally figure out where a single row starts & ends in the file & pick one from the middle of the file & discard the rest....
Redis monitoring
We use Redis for our caching service. Within CiviCRM values retrieved from Redis are 'mostly' cached within the php layer so Redis is hit once per process for most keys. This matters quite a lot for large arrays - as the serialization & unserialization of them has turned out to be slow. In general each process should not be doing `GET` requests for the same key multiple times (unless the cache has been flushed / the key deleted).
I wrote a rather long blog on debugging Redis related slow downs that goes over the next part in more depth
To see what Redis is hitting you can do the following
- Add to your ~/.profile
exportREDIS_PWD=*************
aliasredis-monitor='redis-cli -a $REDIS_PWD monitor'
Note the **** password is in civicrm.settings.php
At this point you can run
redis-monitor>redis.log
If you scp the file down locally there is a script to help make sense of it in the systemtools extension (which might need to be enabled) - e.g
- outputs '`' (backtick) separated file which can be imported into mysql (backtick seems to be otherwise not present so usable but we could make the separator a parameter)
Code cleanups
Preferred Language (completed code cleanup)
Historically our code just added together the language string 'en' and the country string 'DK' to get 'en_DK' - since that wasn't in the database it was just added. so we would love to clean up the language mess.
So we first view civicrm option values from search kit and find out that we have different types of the invalid language options, like typo 1 or a one as xx which represents nothing, or the one added historically.
First we write a drush commend which can filter out the languages that contain the same value and the name ( means they are added to db since wasn't there as en_DK ) then compare with the contacts, that if we do not have any contacts that use this option value, then it's ok to delete it from the civicrm_option_value table.
Initially want to replace them all in db with the civi install, as this ticket, then find we might encounter the issue of deadlock, so then decides to use the cron table.
In order to prevent the db deadlock, we use the cron table with batch size to run the contacts query, and then update the invalid language to the default one, as "update_language."
Used this process control cron job to update the invalid prefer languages for contact based on their current prefer languages as `sh -c "echo '{\"values\":{\"preferred_language\":\"en_US\"},\"where\":[[\"preferred_language\",\"NOT IN\",[\"en_US\", \"en_CA\",\"en_ZA\", \"en_AU\", \"en_GB\", \"en_NZ\", \"en_IN\"]], [\"preferred_language\",\"LIKE\",\"en_%\"]],\"limit\":5000, \"version\":4}' | drush @wmff cvapi Contact.update --in=json"`, refer to this ticket
then we clean up the option values with this patch and run `drush @wmff cvapi WMFDataManagement.CleanInvalidLanguageOptions version=4` to actually delete from civicrm_otpion_value table;
Used this api explorer to update the in_active option value to active: Edit civicrm option value from api explorer, refer to this ticket
eventually, we have the following language options:
MariaDB [civicrm]> select id, value, label, name, is_active, is_default from civicrm_option_value where option_group_id=86 and is_active = 1 order by value;
id
value
label
name
is_active
is_default
586
aa
Afar
aa_ET
1
0
585
ab
Abkhaz
ab_GE
1
0
596
ae
Avestan
ae_XX
1
0
587
af
Afrikaans (af)
af_ZA
1
0
588
ak
Akan
ak_GH
1
0
590
am
Amharic (am)
am_ET
1
0
592
an
Aragonese
an_ES
1
0
591
ar
Arabic (Egypt)
ar_EG
1
0
594
as
Assamese
as_IN
1
0
595
av
Avaric
av_RU
1
0
597
ay
Aymara
ay_BO
1
0
598
az
Azerbaijani (az)
az_AZ
1
0
600
ba
Bashkir
ba_RU
1
0
602
be
Belarusian (be)
be_BY
1
0
608
bg
Bulgarian
bg_BG
1
0
604
bh
Bihari
bh_IN
1
0
605
bi
Bislama
bi_VU
1
0
599
bm
Bambara
bm_ML
1
0
603
bn
Bengali (bn)
bn_BD
1
0
743
bo
Tibetan Standard, Tibetan, Central
bo_CN
1
0
607
br
Breton
br_FR
1
0
606
bs
Bosnian
bs_BA
1
0
610
ca
Catalan; Valencian
ca_ES
1
0
612
ce
Chechen
ce_RU
1
0
611
ch
Chamorro
ch_GU
1
0
617
co
Corsican
co_FR
1
0
618
cr
Cree
cr_CA
1
0
620
cs
Czech
cs_CZ
1
0
704
cu
Old Church Slavonic, Church Slavic, Church Slavonic, Old Bulgarian, Old Slavonic
