civicrm.wikimedia.org
CRM and donations records.
Usage
- This instance went live circa July 2008 and is actively used by our fundraising staff. The instance is used primarily by the fundraising tech staff and the development/major donors staff.
Software
- Apache+PHP
- Drupal: Version 6.17
- CiviCRM: Version 4.1.1
- Jenkins: Version 1.458
Hardware
web frontend
fundraising db cluster
- db1008 @eqiad (master) *active*
- db1025 @eqiad (slave) failover
- storage3 @pmtpa (slave) failover + dumps
- Current databases are 'drupal' for the drupal installation and 'civicrm' for the civicrm installation. Any database names with "dev_" are serving the civicrm-dev.wikimedia.org site
backups
- storage3 & tridge (see backups section below)
Health
- nagios monitors db, general system health
- can our staff run search queries, input data and run reports during normal business hours?
- notifications from various cron and jenkins jobs by mail
Activity
- strace apache process
- turn on extra debugging within php
- turn on civi debug mode
Monitoring
Capacity
- Only vertically scalable for now. Have to get beefier hardware.
Quick Rebuild
- Disable all Jenkins jobs
- Shut down Apache while db maintenance is underway
- Plan A: rerrange the fundraising db cluster and front-end server such that:
- db and frontend are in the same data center (see above) - mysql replication is sane and healthy
- Plan B: restore dumps from one of the backup locations (see above)
- Adjust Civi/Drupal/etc. config files according to the new db cluster arrangement
Dependencies
- CiviCRM depends on the civicrm and drupal databases that live on the fundraising db cluster (see above)
- Also, we're dependent on PayPal messaging us when contributions come in. See overall PayPal system status
- ActiveMQ handles queuing donations from our gateways when a user makes a donation.
- Scripts located in Fundraising.wikimedia.org which manage queue handling for the gateways and our CiviCRM instance
- Hudson manages all of the periodic automated tasks
Start/Restart
Since it's a drupal based module programmed in php running under apache, all that's required is to restart apache.
sudo /usr/sbin/apache2ctl -k graceful
Configuration
/srv/org.wikimedia.civicrm/sites/default/civicrm.settings.php
Typical Problems
- Long running queries
- Kill on DB side
- Paypal Issues
Backups
- storage3:/archive/fundraising - includes nightly database dumps and jenkins build logs
- tridge:/data/fundraising/archive (mirror of storage3's backups) tridge
Escalation Path
- Operations Primary: Jeff Green jgreen@wikimedia.org
- Fundraising Tech Lead: Katie Horn khorn@wikimedia.org
Notes
- There is a development version of CiviCRM available at dev.civicrm.wikimedia.org
- Documentation for the database can be found at Civicrm tables
- We have a watchdog script on the active db that kill any long running queries .. currently it kills anything longer then sixty seconds
- You can find it under /root/kill-slow-queries
References
- CiviCRM core homepage
- Wikimedia CiviCRM
- Wikimedia SVN copy
There is a lightweight Exim4 set up running on the server (see Grosley#Mail_setup for more) which handles sending mail for CiviCRM (via SMTP).
When mail is generated by CiviCRM, Civi stores the mail in some internal queue and does not send the mail until https://civicrm.wikimedia.org/sites/all/modules/civicrm/bin/civimail.cronjob.php is hit. We have scripts living in /home/civicmail/ that handle hitting the appropriate URLs to send and receive mail:
- civimail_send.sh - sends mail for the live site
- dev_civimail_send.sh - send mail for the dev site
- civimail_receive.sh - receives mail for the live site
- dev_civimail_receive.sh - receives mail for the dev site.
Currently, there are two separate Hudson jobs configured to send and receive mail for CiviCRM every 10 minutes.
Periodic Automated Tasks
Hudson handles the periodic automated tasks that need to happen for our CiviCRM set up. This includes:
- Drupal cron.php runs
- Verifying pending PayflowPro transactions
- Consuming donations in the 'donations' queue into CiviCRM (this is handled by drush - note that this particular component needs to be run via drush as an admin user [eg drush --user=1 qc])
- Sending mail for CiviMail
- Receiving mail for CiviMail
Upgrades
See the internal CiviCRM upgrade docs for the latest upgrade information
Public Data Dumps
- Public data dumps are run from srv9 with cron under the reporter user. public_reporting_dump.sh runs fundraising.pl which pulls civicrm data for a given range. These dumps are then sent out to download.wikimedia.org and publicly consumed.
Relevant files are under the private svn depot under
svn.wikimedia.org/svn-private/wmf/donation/fundraising.pl svn.wikimedia.org/svn-private/wmf/donation/crons/public_reporting_dump.sh
Issues
- Maintained on the CiviCRM wiki