CiviCRM requires a 'host CMS' and to that end we use Drupal7. Drupal 7 is EOL in November 2022 and next year we plan to upgrade to Drupal 9 - or maybe even 10. Our goal, however, is that we do not use any CMS-specific code going forwards. While we currently expect to stick with Drupal in we should be equally able to move to Wordpress. To this end we are in the process of migrating our drupal modules to CiviCRM extensions.
drush
Drush is a really useful drupal command line utility. There is a lot of documentation about drush on the internet but a few things to know with regards to WMF.
On production, staging and in our docker dev set up we have an alias 'wmff' which tells drush details about where the code is and to use user 1.
Common usage:
Command
Where
What
`drush @wmff updb`
local dev and prod
Run any database updates that need to be run
`drush @wmff up --security-only`
local dev
Download and install any security updates (these are then checked into git to deploy)
If you see problems finding classes while enabling the civicrm extension, this may be due to stale metadata cached in redis. You can use the queues-redis-clear.sh script to clear that out (note you will also lose all messages in the donation and job queues).
Log Tables
We use CiviCRM logging to create log tables which we use for
- Threshold queries - query log_civicrm_contribution for contributon updates in the last 5 minutes (not includes any sort of update)
- Forensics - eg https://phabricator.wikimedia.org/T311438 involved figuring out history of a contact around 18 months ago. Importantly determined they never had any contributions - ie didn't have them & then they were deleted, but never existing
Or best practices
CiviCRM uses the drupal whitespace standards (sadly adopted before the world co-alesced on something different) so all drupal modules & civicrm extensions (including the CiviCRM codebase) should use that. When using phpstorm installing the drupal plugin will add this standard. Within the vendor directory PSR2 is likely more appropriate
We agreed to name our civicrm extensions with the prefix 'wmf-` where they are WMF specific and just the name if they are to be shared - ie `wmf-civicrm` and `deduper` respectively. This applies going forwards (we may or may not fix existing ones)
Running CiviCRM behind a firewall
In order to run CiviCRM behind a firewall we have disabled some SystemChecks on production (but not on our local developer environments) and use a ckeditor image uploader that embeds images. These are documented in the main CiviCRM firewall documentation. In our case the use of browser certificates for authentication means the server cannot reach it's own urls during crons
Custom fields in CiviCRM can be created through the user interface. In order to allow flexibility to our users the arrangement we have with our super-users (Nora, Rosie) is that they can create custom fields through the UI but they should create a phab task so that fr-tech can follow up ( add field to advance search and so on ).
The follow up by frtech is in 2 parts - ensuring the fields are present in our dev environments and creating triggers.
Ensuring the fields are present in our dev environments
Keeping our dev environment fields in sync is a best-efforts endeavor rather than something we keep 100% in sync, but it does make it easier for us to develop locally. There is currently an 'old method' and a 'new method. The new method is to use the CiviCRM managed entities functionalityas reflected in `CustomGroupMatchingGiftInformation.mgd.php` file. The previous barriers to this standardised approach, around the use of the more efficient BulkSave action and matching existing entities, are now resolved. However we DO set the action on update & cleanup to 'never' to avoid anything unexpected on production. The command to reconcile these is
drush @wmff cvapi Managed.reconcile version=4
The old method involves tracked/synced fields being declared in the CustomGroups.php file in the Managed directory in the `wmf_civicrm` extension. '. This file follows the conventions of the CiviCRM managed entities functionality and the fields declared in the file are added to our developer builds on install.
However, because the file is not a direct match to the Custom fields on prod, we have not registered this file with the civicrm_managed hook, and instead we have a custom wmf command which adds any declared fields in the CustomGroups.php, that are missing for the dev site.
This command adds CustomGroups and CustomFields to dev sites if missing, but does not update them. It only creates option values if the field did not previously exist or it is being run in a development environment - ie we want to add but not update on live.
Note that
When declaring the fields in the CustomGroups array an easy way to get the data from live is to use the API v4 explorer - once the criteria are selected & execute has been hit the field data is listed in a json format and there is even an option next to it to switch to a php format. Fields that do not differ from the defaults (including is_active) should be removed from the resulting array, along with the id field. The explorer can be used in a similar way to get the CustomFields in the group and any option values (using the option_group_id from the custom field definition). Do not include option_group_id or custom_group_id in the checked in array
In CivICRM all field types can be extended with custom groups - however, CiviCRM must know that they can be extended. CiviCRM has a hard coded mapping of the common entities (Contribution, Contact etc) but also maintains an option group ` cg_extends` with other entities. When extending an entity type that is not extendable by default we need to ensure the option value exists.
In some cases the functionality of the custom fields are owned by extensions rather than WMF user driven. In these cases the fields are declared in the relevant extension (e.g the Omnimail extension installs 2 custom groups and the relationship block extension installs one). These are written into the Upgrade classes in the relevant extensions and extension upgrades are run using the following command.
drush @wmff cvapi Extension.upgrade
Update triggers on production
We use mysql triggers to log civicrm database updates to the log tables. These triggers are managed by CiviCRM. However, our production user does not have enough mysql permissions to create the triggers within mysql. To get around this we use a CiviCRM setting on production to log the sql to update the triggers to a file rather than live update them. We then check this file into our crm repo (sites/all/modules/wmf_civicrm/scripts/triggers.mysql currently) and fr-Ops run the file on live.
On development environments triggers are automatically updated in the database - which is generally easier - but to make your local output the triggers as live does the logging_no_trigger_permission setting can be enabled
Trigger generation needs to be done on production as the fields differ slightly on staging / dev environments. There are a few methods but turning logging off & back on generally generates consistent output. ie
This will generate a file named something like CiviCRM.trigger62451ae5ab5a5mYm67702126718965e4a41105a08d6202e60.sql that will be in drupal/sites/default/files/civicrm/ConfigAndLog/ - copy this back to your home drive and scp it back to your local machine as sites/all/modules/wmf_civicrm/scripts/triggers.mysql. Changes to this files are committed, reviewed and deployed but they will not be 'live' until fr-tech-ops loads them so once deployed they need to be engages to run the latest triggers.mysql file
Automated emails from CiviCRM
We send out the following automated emails
Recurring failure notifications - these are send when a monthly recurring email is failing and encourages people to set up a new one. It is not sent if they have an active recurring email.
Thank you letters - these are send by an automated job for every donation in CiviCRM, unless the 'no_thankyou' field is populated or it is a recurring donations
End of year emails - these are sent at the start of the year to cover all the recurring contributions in the previous year. These can also be sent ad hoc to individual donors (in which case they include all donations, not just the recurring ones)
You can run the following drush command on dev or staging to import (60) contributions and get timings.
drush @wmff qperf-d 60"your comment"
If you wish to see what queries run you can add this line to your civicrm.settings.php file - the 'n' is just part of the file name, in case you want to do separate runs
docker@civicrm:/srv/civi-sites/wmff$ ls -altr drupal/sites/default/files/civicrm/ConfigAndLog/
total 27816
-rw-r--r-- 110001000202 Dec 222021 .htaccess
drwxr-xr-x 8100010004096 Mar 802:51 ..
-rw-r--r-- 110001000215636 Jun 3003:50 CiviCRM.b4f78a4e4ce41be7806fadbb.log.202207050615
-rw-r--r-- 110001000623 Jul 900:41 CiviCRM.b4f78a4e4ce41be7806fadbb.log.202208030206
-rw-r--r-- 11000100031549 Aug 2323:11 CiviCRM.b4f78a4e4ce41be7806fadbb.log
drwxr-xr-x 2100010004096 Aug 2506:14 .
-rw-r--r-- 11000100028205254 Aug 2506:14 CiviCRM.sql_logn.b4f78a4e4ce41be7806fadbb.log
The extension org.wikimedia.systemtools provides a script / api to help analyse this file - you may need to installl it - in the UI it is called "Home for WMF helpers".
Once enabled you can run the following
It will output a csv with a cleaned up version of the queries to the directory you are in - the output will give the filename (spoiler - it's gonna be query_log_parsed.csv).
Also note the extension has a Readme.md....
At this point I generally figure out where a single row starts & ends in the file & pick one from the middle of the file & discard the rest....
Redis monitoring
We use Redis for our caching service. Within CiviCRM values retrieved from Redis are 'mostly' cached within the php layer so Redis is hit once per process for most keys. This matters quite a lot for large arrays - as the serialization & unserialization of them has turned out to be slow. In general each process should not be doing `GET` requests for the same key multiple times (unless the cache has been flushed / the key deleted). To see what Redis is hitting you can do the following
- Add to your ~/.profile
exportREDIS_PWD=*************
aliasredis_monitor='redis-cli -a $REDIS_PWD monitor'
Note the **** password is in civicrm.settings.php
At this point you can run
redis-monitor > redis.log
If you scp the file down locally there is a script to help make sense of it in the systemtools extension (which might need to be enabled) - e.g
- outputs '`' (backtick) separated file which can be imported into mysql (backtick seems to be otherwise not present so usable but we could make the separator a parameter)
Preferred Language
Historically our code just added together the language string 'en' and the country string 'DK' to get 'en_DK' - since that wasn't in the database it was just added. so we would love to clean up the language mess.
So we first view civicrm option values from search kit and find out that we have different types of the invalid language options, like typo 1 or a one as xx which represents nothing, or the one added historically.
First we write a drush commend which can filter out the languages that contain the same value and the name ( means they are added to db since wasn't there as en_DK ) then compare with the contacts, that if we do not have any contacts that use this option value, then it's ok to delete it from the civicrm_option_value table.
Initially want to replace them all in db with the civi install, as this ticket, then find we might encounter the issue of deadlock, so then decides to use the cron table.
In order to prevent the db deadlock, we use the cron table with batch size to run the contacts query, and then update the invalid language to the default one, as "update_language."
Used this process control cron job to update the invalid prefer languages for contact based on their current prefer languages as `sh -c "echo '{\"values\":{\"preferred_language\":\"en_US\"},\"where\":[[\"preferred_language\",\"NOT IN\",[\"en_US\", \"en_CA\",\"en_ZA\", \"en_AU\", \"en_GB\", \"en_NZ\", \"en_IN\"]], [\"preferred_language\",\"LIKE\",\"en_%\"]],\"limit\":5000, \"version\":4}' | drush @wmff cvapi Contact.update --in=json"`, refer to this ticket
then we clean up the option values with this patch and run `drush @wmff cvapi WMFDataManagement.CleanInvalidLanguageOptions version=4` to actually delete from civicrm_otpion_value table;
Used this api explorer to update the in_active option value to active: Edit civicrm option value from api explorer, refer to this ticket
eventually, we have the following language options:
MariaDB [civicrm]> select id, value, label, name, is_active, is_default from civicrm_option_value where option_group_id=86 and is_active = 1 order by value;
id
value
label
name
is_active
is_default
586
aa
Afar
aa_ET
1
0
585
ab
Abkhaz
ab_GE
1
0
596
ae
Avestan
ae_XX
1
0
587
af
Afrikaans (af)
af_ZA
1
0
588
ak
Akan
ak_GH
1
0
590
am
Amharic (am)
am_ET
1
0
592
an
Aragonese
an_ES
1
0
591
ar
Arabic (Egypt)
ar_EG
1
0
594
as
Assamese
as_IN
1
0
595
av
Avaric
av_RU
1
0
597
ay
Aymara
ay_BO
1
0
598
az
Azerbaijani (az)
az_AZ
1
0
600
ba
Bashkir
ba_RU
1
0
602
be
Belarusian (be)
be_BY
1
0
608
bg
Bulgarian
bg_BG
1
0
604
bh
Bihari
bh_IN
1
0
605
bi
Bislama
bi_VU
1
0
599
bm
Bambara
bm_ML
1
0
603
bn
Bengali (bn)
bn_BD
1
0
743
bo
Tibetan Standard, Tibetan, Central
bo_CN
1
0
607
br
Breton
br_FR
1
0
606
bs
Bosnian
bs_BA
1
0
610
ca
Catalan; Valencian
ca_ES
1
0
612
ce
Chechen
ce_RU
1
0
611
ch
Chamorro
ch_GU
1
0
617
co
Corsican
co_FR
1
0
618
cr
Cree
cr_CA
1
0
620
cs
Czech
cs_CZ
1
0
704
cu
Old Church Slavonic, Church Slavic, Church Slavonic, Old Bulgarian, Old Slavonic
