civicrm.wikimedia.org

From Wikitech
Jump to: navigation, search

CRM and donations records.

Usage

  • This instance went live circa July 2008 and is actively used by our fundraising staff. The instance is used primarily by the fundraising tech staff and the development/major donors staff.

Software

  • Apache+PHP
  • Drupal: Version 6.17
  • CiviCRM: Version 4.1.1
  • Jenkins: Version 1.458

Hardware

web frontend

fundraising db cluster

  • db1008 @eqiad (master) *active*
  • db1025 @eqiad (slave) failover
  • storage3 @pmtpa (slave) failover + dumps
    • Current databases are 'drupal' for the drupal installation and 'civicrm' for the civicrm installation. Any database names with "dev_" are serving the civicrm-dev.wikimedia.org site

backups

  • storage3 & tridge (see backups section below)

Health

  • nagios monitors db, general system health
  • can our staff run search queries, input data and run reports during normal business hours?
  • notifications from various cron and jenkins jobs by mail

Activity

  • strace apache process
  • turn on extra debugging within php
  • turn on civi debug mode

Monitoring

Capacity

  • Only vertically scalable for now. Have to get beefier hardware.

Quick Rebuild

  1. Disable all Jenkins jobs
  2. Shut down Apache while db maintenance is underway
  3. Plan A: rerrange the fundraising db cluster and front-end server such that:
 - db and frontend are in the same data center (see above)
 - mysql replication is sane and healthy
  1. Plan B: restore dumps from one of the backup locations (see above)
  2. Adjust Civi/Drupal/etc. config files according to the new db cluster arrangement

Dependencies

  • CiviCRM depends on the civicrm and drupal databases that live on the fundraising db cluster (see above)
  • Also, we're dependent on PayPal messaging us when contributions come in. Overall PayPal system status can be found here
  • ActiveMQ handles queuing donations from our gateways when a user makes a donation.

Start/Restart

Since it's a drupal based module programmed in php running under apache, all that's required is to restart apache.

sudo /usr/sbin/apache2ctl -k graceful

Configuration

/srv/org.wikimedia.civicrm/sites/default/civicrm.settings.php

Typical Problems

Backups

  • storage3:/archive/fundraising - includes nightly database dumps and jenkins build logs
  • tridge:/data/fundraising/archive (mirror of storage3's backups) tridge

Escalation Path

  • Operations Primary: Jeff Green jgreen@wikimedia.org
  • Fundraising Tech Lead: Katie Horn khorn@wikimedia.org

Notes

  • There is a development version of CiviCRM available at dev.civicrm.wikimedia.org
  • Documentation for the database can be found at Civicrm tables
  • We have a watchdog script on the active db that kill any long running queries .. currently it kills anything longer then sixty seconds
    • You can find it under /root/kill-slow-queries

References

Mail

There is a lightweight Exim4 set up running on the server (see Grosley#Mail_setup for more) which handles sending mail for CiviCRM (via SMTP).

When mail is generated by CiviCRM, Civi stores the mail in some internal queue and does not send the mail until https://civicrm.wikimedia.org/sites/all/modules/civicrm/bin/civimail.cronjob.php is hit. We have scripts living in /home/civicmail/ that handle hitting the appropriate URLs to send and receive mail:

  • civimail_send.sh - sends mail for the live site
  • dev_civimail_send.sh - send mail for the dev site
  • civimail_receive.sh - receives mail for the live site
  • dev_civimail_receive.sh - receives mail for the dev site.

Currently, there are two separate Hudson jobs configured to send and receive mail for CiviCRM every 10 minutes.

Periodic Automated Tasks

Hudson handles the periodic automated tasks that need to happen for our CiviCRM set up. This includes:

  • Drupal cron.php runs
  • Verifying pending PayflowPro transactions
  • Consuming donations in the 'donations' queue into CiviCRM (this is handled by drush - note that this particular component needs to be run via drush as an admin user [eg drush --user=1 qc])
  • Sending mail for CiviMail
  • Receiving mail for CiviMail

Upgrades

See the internal CiviCRM upgrade docs for the latest upgrade information

Public Data Dumps

  • Public data dumps are run from srv9 with cron under the reporter user. public_reporting_dump.sh runs fundraising.pl which pulls civicrm data for a given range. These dumps are then sent out to download.wikimedia.org and publicly consumed.

Relevant files are under the private svn depot under

  svn.wikimedia.org/svn-private/wmf/donation/fundraising.pl
  svn.wikimedia.org/svn-private/wmf/donation/crons/public_reporting_dump.sh

Issues