Jump to content


From Wikitech

This page documents to the 'frqueue' redis service which handles contribution tracking messages and donor preference change messages. This service runs on the 'frqueue' server role in the Fundraising server cluster.


  • We run redis basic replication
  • There are two redis instances running as separate services
    • redis-frqueue (tcp/6379)
    • redis-donor-prefs (tcp/6380)
  • The redis-frqueue instance uses key "sequence_contribution-tracking" as a sequence generator for contribution tracking.
  • When switching origin servers, shut down the old origin server before making replication changes to be sure it doesn't receive any new messages after replication is changed.
  • You can invoke redis commands from a shell using redis-cli, or you can log in via redis-cli and run commands within in a client.

Origin Swap

Confirm that all client services (paymentswiki, civicrm queue consumers, smashpig, etc) are shut down.

Obtain the redis masterauth passwords from /etc/redis/*.conf

If the old redis origin server is dead, verify the new origin server has the latest Contribution ID.

  • The last ID imported by queue consumers can be obtained from the drupal database.
SELECT id,contribution_id FROM drupal.contribution_tracking WHERE contribution_id IS NOT NULL ORDER BY id DESC LIMIT 10;
  • Use redis-client to compare the autoincrement counter to the latest contribution ID.
# redis-cli> AUTH <password>> GET sequence_contribution-tracking> exit


  • If replication died in some wierd way, it's possible different replicas have different autoincrement values. In this case choose the one with the greatest value as the new origin.
  • If contribution ID from the database is higher than the redis autoincrement, there is a risk of corrupting the database if you don't adjust the redis autoincrement. FR Tech should be consulted for how to proceed.

Promote new origin server for redis-frqueue

  • On the new queue server stop replication, retaining dataset, become origin. Check if things look ok.
# redis-cli> AUTH <password>> REPLICAOF NO ONE> INFO> GET sequence_contribution-tracking> exit

# redis-cli -p 6380> AUTH <password>> REPLICAOF NO ONE> INFO> exit

On each replica, change origins (discarding dataset) and resynchronize, check status.

# redis-cli> AUTH <password>> REPLICAOF <new redis origin FQDN> 6379> INFO> exit

# redis-cli -p 6380> AUTH <password>> REPLICAOF <new redis origin FQDN> 6380> INFO> exit

This adjusts puppetized clients including civicrm, payments-listener, and sets redis replication configuration to survive a service restart.

  • edit (frack) puppet/hieradata/site/common.yaml
queue_redis_primary: '<new redis origin FQDN>'
  • git commit, git push, puppet-merge