This page contains historical information. It may be outdated or unreliable.

CRM and donations records.

Usage

This instance went live circa July 2008 and is actively used by our fundraising staff. The instance is used primarily by the fundraising tech staff and the development/major donors staff.

Software

Apache+PHP
Drupal: Version 6.17
CiviCRM: Version 4.1.1
Jenkins: Version 1.458

Hardware

web frontend

aluminium @eqiad *active*
grosley @pmtpa standby

fundraising db cluster

db1008 @eqiad (master) *active*
db1025 @eqiad (slave) failover
storage3 @pmtpa (slave) failover + dumps
- Current databases are 'drupal' for the drupal installation and 'civicrm' for the civicrm installation. Any database names with "dev_" are serving the civicrm-dev.wikimedia.org site

backups

storage3 & tridge (see backups section below)

Health

nagios monitors db, general system health
can our staff run search queries, input data and run reports during normal business hours?
notifications from various cron and jenkins jobs by mail

Activity

strace apache process
turn on extra debugging within php
turn on civi debug mode

Monitoring

Nagios

Capacity

Only vertically scalable for now. Have to get beefier hardware.

Quick Rebuild

Disable all Jenkins jobs
Shut down Apache while db maintenance is underway
Plan A: rerrange the fundraising db cluster and front-end server such that:

 - db and frontend are in the same data center (see above)
 - mysql replication is sane and healthy

Plan B: restore dumps from one of the backup locations (see above)
Adjust Civi/Drupal/etc. config files according to the new db cluster arrangement

Dependencies

CiviCRM depends on the civicrm and drupal databases that live on the fundraising db cluster (see above)

Also, we're dependent on PayPal messaging us when contributions come in. See overall PayPal system status

ActiveMQ handles queuing donations from our gateways when a user makes a donation.

Scripts located in Fundraising.wikimedia.org which manage queue handling for the gateways and our CiviCRM instance

Hudson manages all of the periodic automated tasks

Start/Restart

Since it's a drupal based module programmed in php running under apache, all that's required is to restart apache.

sudo /usr/sbin/apache2ctl -k graceful

Configuration

/srv/org.wikimedia.civicrm/sites/default/civicrm.settings.php

Typical Problems

Long running queries
- Kill on DB side
Paypal Issues

Backups

storage3:/archive/fundraising - includes nightly database dumps and jenkins build logs
tridge:/data/fundraising/archive (mirror of storage3's backups) tridge

Escalation Path

Operations Primary: Jeff Green jgreen@wikimedia.org
Fundraising Tech Lead: Katie Horn khorn@wikimedia.org

Notes

There is a development version of CiviCRM available at dev.civicrm.wikimedia.org
Documentation for the database can be found at Civicrm tables
We have a watchdog script on the active db that kill any long running queries .. currently it kills anything longer then sixty seconds
- You can find it under /root/kill-slow-queries

References

CiviCRM core homepage
Wikimedia CiviCRM
Wikimedia SVN copy

Mail

There is a lightweight Exim4 set up running on the server (see Grosley#Mail_setup for more) which handles sending mail for CiviCRM (via SMTP).

When mail is generated by CiviCRM, Civi stores the mail in some internal queue and does not send the mail until https://civicrm.wikimedia.org/sites/all/modules/civicrm/bin/civimail.cronjob.php is hit. We have scripts living in /home/civicmail/ that handle hitting the appropriate URLs to send and receive mail:

civimail_send.sh - sends mail for the live site
dev_civimail_send.sh - send mail for the dev site
civimail_receive.sh - receives mail for the live site
dev_civimail_receive.sh - receives mail for the dev site.

Currently, there are two separate Hudson jobs configured to send and receive mail for CiviCRM every 10 minutes.

Periodic Automated Tasks

Hudson handles the periodic automated tasks that need to happen for our CiviCRM set up. This includes:

Drupal cron.php runs
Verifying pending PayflowPro transactions
Consuming donations in the 'donations' queue into CiviCRM (this is handled by drush - note that this particular component needs to be run via drush as an admin user [eg drush --user=1 qc])
Sending mail for CiviMail
Receiving mail for CiviMail

Upgrades

See the internal CiviCRM upgrade docs for the latest upgrade information

Public Data Dumps

Public data dumps are run from srv9 with cron under the reporter user. public_reporting_dump.sh runs fundraising.pl which pulls civicrm data for a given range. These dumps are then sent out to download.wikimedia.org and publicly consumed.

Relevant files are under the private svn depot under

  svn.wikimedia.org/svn-private/wmf/donation/fundraising.pl
  svn.wikimedia.org/svn-private/wmf/donation/crons/public_reporting_dump.sh

Issues

Maintained on the CiviCRM wiki