Fundraising/Team processes/Onboarding

From Wikitech

Accounts & Privileges to get

Prep

  • Write your name inside the team contact list with phone number and email address, this will add you to Gerrit as project's git reviewer.
  • Get IRC join team #wikimedia-fundraising and #xxxx(ask your team member). Choose your own irc connection methods, limechat or irccloud or online works
  • Manage gmail filters
  • Add a team calendar for online meetings.
  • Know more about FR team & Fundraising, with lots links for payments and test links
  • Install your own IDE, JetBrains tool is free with wiki email if you use phpStorm.
  • Join Asana with gmail and join team “online fundraising” by searching coworker’s email.
  • Check Fr-tech priorities meeting notes for each spring's task overview

Set up local Env

  1. Refer to the Readme, get all the code we are working on in local env
  2. install git-review for peer review
  3. use Zuul to check code reveiw by Jenkins.
  • Database
  1. use any GUI you like, with the Docker up, get all the code and the database, Redis and so on based on containers
  2. if use the MysqlWorkBench, set useSSL=0 to advance others to enable no ssl for standard TCP/IP.
  • SSH
  1. Yubikey will be requested by your manger.
  2. With the Yubikey set up, able to go to the dev server with ssh access,
  3. To set up your Yubikey, you needs to be corporate with Dallas
  4. With ssh access, you can perform deploy(see below, ask peers to do it together at your first time.
  1. We have three projects (Smashpig, payments, civicrm) which smashipig is working for both payments and civicrm.
  2. Whenever we has some package version updates, need to update the correspondent package version for donor interface and smashpig
  3. the access could be granted by Elliott

Some account might not need at this moment but good to have later

Your first few days/weeks

Day 1

  • Participate the Onboarding meeting and set up some accounts
  • Get to know the team and set up some 1:1 with your onboard buddy
  • Get accesses per above list

Day 2

What is Gerrit and get to know code in general

  1. Register Gerrit and get your username, use it to download code from Gerrit and run setup.sh with Gerrit username as reviewer name to set docker up;
  2. After you got the yubikey, you can pull the config-private from frpm1002:/var/lib/git/fundraising-dev-private.git, also for this folder, we use git push instead of git review
  3. All the live version config files can be found from server 
    yourname@frdev1002:/srv/www/org/wikimedia/listeners/SmashPig/local-config$ ls -a
    set up your local env and start to get to know the code

Day 3

Get familiar with the payment process

  • After install docker successfully, go to https://localhost:9001 (if not able to open localhost in https with Chrome, enable this chrome://flags/#allow-insecure-localhost
  • Make test payment with adyen or ingenico test card, it will perform auth first, in local env, we need to manually check payment from Redis, then use queue testing doc to consume queue to db to actually capture payments.
./queues-redis-cli.sh  (this will log you in redis)
Keys * (this will show all the keys)
lrange donations 0 -1 (this will show all the jobs in donations queue)
cd <roor dir> (go back to root dir)
./civicrm-drush.show (this will logi you in civi)
drush --user-1 -v -r /srv/civi-sites/wmff/drupal qc (run queue consume to exec the queue to db)
docker@civicrm:/srv/civi-sites/wmff/drupal$ drush --user=1 -v afqc (this is consume fraud contributions)

After consume the queue, we will be able to see the transaction from

  1. CiviCrm
  2. CiviCrm database
  3. payment method's back office with payment details.

How to monitor production log

ssh frlog1002
cd /var/log/remote
tail -f payments-ingenico (Check out how many ingenico payments)
tail -f payments-listener-smashpig (Check out payments in general which smashpig handle auth)
grep <keyword> (to find the keyword from log)

What is Irc

  • Register your nickname
  • Set up cloak

Add google calendar and know agile meetings

  • Daily Autopsy (to check if there is any blocker)
  • Backlog Refinement (repeat every week to clear the backlog)
  • Spring Review, retro and planing ( Repeat every 2 weeks, to plan what should we do in next spring)
    • What is good this spring and what could be better with retro notes or miro dashboard
    • Review if last spring been cleared and are those tasks
    • Estimate story point for last spring's done issues
    • Vote for the name of next spring
    • Adding issues for next spring from triage
  • Talk Talk (Solve blocker together or sync up recent issues, one repeat every week and one repeat every two weeks)
  • Analytics sync up (repeat every 2 weeks to make sure we are sync in each Quarters)
  • Civi Fortnightly (Talk about Civi with whole advancement team, repeat every 2 weeks)
  • Alerts Retro (text based meeting to look at recent Failmail and alerts, repeat every 2 weeks)
  • some 1:1 meetings

Day 4

Get familiar with local test process:

  • A: queue’s testing
  • B: queue Message
  • c: Make payment with Redis queue monitor to observe all the transaction steps
    cd <rootdir>
./queues-rediscli.sh MONITOR
    If smashpig has too many pending transactions, which might due to fraud, then clear them with this cmd,
    docker-compose exec smashpig bash (go inside docker server first)
cd /srv/smashpig/  (go to the right dir )
php Maintenance/ConsumePendingQueue.php (this put all fraud into pending table first)
php Maintenance/DeleteExpiredPendingMessages.php --gateway=ingenico --days=1
    To clear the pending transactions with a certain gateway and exist for more than 1 days
  • clear drush cache and system flush to clear any cache, then run the PendingTable.consume also in local source code open xdebug.ini inside config civicrm, update xdebug.remote_autostart=on then start debug.
    docker@civicrm:/srv/civi-sites/wmff/drupal$ drush cc
docker@civicrm:/srv/civi-sites/wmff/drupal$ drush cvapi System.flush
docker@civicrm:/srv/civi-sites/wmff/drupal$ drush cvapi PendingTable.consume version=4 gateway=ingenico

Day 5

Understand each dev test URLs

  • Payments URL: https://localhost:9001       payments forms
  • Payments http URL: http://localhost:9009      same as above, but over http (used to receive requests via the routable URL)
  • Payments test routable URL: https://paymentstest1.wmcloud.org (see README.md)   For Apple Pay testing
  • WMF CiviCRM install URL: https://wmff.localhost:32353/civicrm CiviCRM with all of our drupal modules and custom Civi modules
  • Generic CiviCRM install (based on upstream master) URL: https://dmaster.localhost:32353/civicrm  Core CiviCRM
  • Civicrm user/password: admin/admin
  • Mailcatcher - mails sent from CiviCRM core code: http://localhost:1080  For civi, email display like the EOY
  • CiviProxy URL: https://localhost:9005    do not need to worry about this now
  • SmashPig IPN listener routable URL: https://paymentsipntest1.wmcloud.org (see README.md) test adyen apple pay
  • E-mail Preference Center URL: https://localhost:9002/index.php/Special:EmailPreferences    not working on that
  • PrivateBin read-only URL: https://localhost:9007         fancy page for contributor to see their activities
  • PrivateBin read-write URL: https://localhost:9008        donor activities with write access

Git review

  • install git-review
  • submitting changes with git-review
    • Git pull from master
    • Git checkout -b <newbrach>
    • Make modification to the code, then use git status to check which code is been modified and will we add it or use ui to add certain change files
    • Git add <right paths of the code file>
    • Git commit
      WIP:<comment>

<more desc for this change>

Bug: <Gerrit Tag Name>
      if not satisfied, then
    • Git commit –amend (if your want to update exit with :wq)
    • Git review (this part submit the code for review and auto run by Jenkins with +2  if did not break anything, then all the commit will automatically linked with Gerrit and Phabricator).
    • Note: you can always reply "recheck" to your commit to let jenkins run the check again
  • Modify you commit
    • Beside the code you commit, you will find a button as "UPDATE CHANGE"
    • Click that one, then you can find the 4 steps instruction as
      1. Checkout this change locally and make your desired modifications to the files: `git fetch https://gerrit.wikimedia.org/r/xxxxxx && git checkout FETCH_HEAD
      2. Update the local commit with your modifications using the following command: `git add . && git commit --amend --no-edit` (Leave the "Change-Id:" line of the commit message as is)
      3. Push the updated commit to Gerrit: `git push origin HEAD:refs/for/master`
      4. Refresh this page to view the the update.
  • Review other’s commits
    • Go to the Gerrit dashboard and find the patches you want to review
    • git-review -d <tagName> to checkout that branch or from the Gerrit ui, click download, use anonymous http to checkout the new branch or just checkout to the patch branch.
    • If everything is ok, leave a comment below of the Phabricator task and then use Zuul to run reviewed code with Jenkins. If passed, then +2 this code is ready to move to production.
    • During debug, could use export PHP_IDE_CONFIG="serverName=civi"to enable xDebug for phpStorm (where depending on your IDE you may wish to set the environment variable PHP_IDE_CONFIG.For phpstorm it should hold a value like "serverName=wmff" - where wmff is configured as a deployment server in phpStorm. If you are using drush then the site alias will set this variable - e.g drush @wmff will run drush in the context of the wmff.localhost site, including the PHP_IDE_CONFIG.serverName key)

Day 6

Update package at Packagist

  • 1: Login to root dir where you have docker code, fundraising handle three main repos:

donation-interface

cd src/payments/extensions/DonationInterface
git remote -v   (if return https then convert to ssh)
git remote set-url origin ssh://<reviwer>@gerrit.wikimedia.org:29418/mediawiki/extensions/DonationInterface
git pull (run git remote -v again to check if remote set to ssh then pull the newest code, handle all the payment methods)

smash-pig

cd ~/<rootdir>/src/smashpig
git pull  (to get the most updated version of smaship which handle both civicrm -when donor is doing recurring payment and payments flow)

CiviCRM

cd ~/<rootdir>/src/civi-sites/wmff
git pull  (get the most updated version of civicrm which manage the customer backoffice)
git remote

Upgrade drupal 8 to 9 in year 2020, and in the future remove drupal just save civi

In wmff > drupal > sites>  all  (those are the old drupal module)

Git remote should be /crm

  • 2: update tag to Packagist if necessary: (after pull most updated code from master, we have two packages which are wikimedia/smash-pig and wikimedia/donation-interface)
    1. git tag (to check current tag)
    2. git tag v.x.x.x.x (to mark current repo to this tag version)
    3. git push --tags (to push new tag version)
    4. Go to Packagist, login, find the package, then hit update
  • 3: update packages then update it in code:
    1. docker-compose exec civicrm bash  (go to local civicrm server)
    2. cd src/civi-sites/wmff   (go to wmff repo)
    3. composer self-update (if necessary)
    4. composer update wikimedia/donation-interface wikimedia/smash-pig (must update them at the same time)
    5. Update the new tag version number in DonationInterface composer.json

Update smash-pig version for civi

(since civi use both di and smashpig for paypal, we need to have both di and smash-pig update at the same time):

PART I: SMASH-PIG TAG UPDATE

  1. Go to Packagist, check latest SmashPig version
  2. Checkout latest origin/master in your local copy of the SmashPig repo
  3. Update the git tag to later one from Packagist
    git tag v0.8.2.11 HEAD
git push origin v0.8.2.11
  4. Go to wikimedia/smash-pig - Packagist
  5. Click the green 'Update' button

PART II: DI update based on new smash-pig version

  1. Checkout to payments, confirm the branch is fundraising/REL1_35, pull latest
  2. Go into  /payments/extensions/DonationInterface, pull latest master then modify /payments/extensions/DonationInterface/composer.json composer.json to for smash-pig with the new version
    git add composer.json
git commit
git review
    Example: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/833109/
  3. After pr merged,  go back to /payments, delete the old vendor folder
    rm -rf vendor
  4. Generate a new vendor folder based on new smash-pig version
    git submodule update --init vendor
    Make sure vendor also use the master branch
    cd vendor
git checkout fundraising/REL1_35
cd ../
  5. from /payments update vendor
    composer update --no-dev wikimedia/smash-pig
  6. Go into /payments/vendor (git remote -v should be mediawiki/vendor, also branch should be fundraising/REL1_35) , you will see the smash-pig update added to vendor
    git add .
git commit
git review
    Example:https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/833115
  7. Make sure the vendor commit is merged in gerrit
  8. if unblock civi can skip deployment (if want do deployment: follow Fundraising/Cluster/Deployments - Wikitech)
  9. Back to DI, submit the newer version for DI to packagist
    git tag v0.5.7.10 HEAD
git push origin v0.5.7.10
    then go to wikimedia/donation-interface - Packagist
  10. Click the green 'Update' button

PART III: CIVI update both donation-interface  and smash-pig version

  1. Git checkout latest /civi-sites/wmff locally, then update the composer.json with newest di and smash-pig version to composer.json
  2. Go to the civi box, update both smash-pig and di
    composer update --no-dev wikimedia/smash-pig wikimedia/donation-interface
  3. From local wmff, if only update for smash-pig, no need to add extensions/DonationInterface
    git add composer.json composer.lock
git commit
git review
    Example: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/831146/ (sometimes if got civi conf, just add the commit together with the composer.json and lock file)
  4. Once above merge, can rebase the blocked civi pr.

Day 7

Understand redis queue

./queues-redis-cli.sh (login to redis)
Keys * (check all the keys in redis)

1: contribution-tracking  -see under current session, actual payment tried how many times with detail

2: payments-init -payment in pending before it finished

3: sequence_contribution-tracking  -hold the number of contribution tired

4: payments-antifraud  -contain all fraud transactions

5: pending -not completed

6: donations  - all complete, user redirect to thank you page (this is the donation we just made in json)

you can either use Redis desktop manager to view key value pairs or just use lrange <keyName> 0 -1 to view value with keyName.

Day 8

Lint for JS

  • Payment DonationInterface uses the mediawiki rules:
    • Under /src/payment/extension/DonationInterface Run `npm install -D stylelint-config-wikimedia`, then you will find your esLint enabled
  • CiviCrm is using civilint rule, also here is the doc civi has for phpStorm config.

Code style for PHP

XDEBUG for phpStorm

  • Go to Preferences -> PHP -> Servers, add three servers with
  1. payment-wiki:
    • path mapping: /src/payment <--> /var/www/html;
    • host: localhost;
    • port: 9001;
    • Debuger: Xdebug.
  2. civi:
    • path mapping: /src/citi-sites <--> /srv/citi-sites;
    • host: wmff.localhost;
    • port: 32353;
    • Debuger: Xdebug.
  3. smashpig:
    • path mapping: /src/smashpig <--> /srv/smashpig;
    • host: localhost;
    • port: 9006;
    • Debuger: Xdebug.
  • Edit configuration -> add PHP Remote Debug -> check Filter debug connection by IDE key
    • Payment-wiki:
      • Server: payment-wiki
      • IDE Key: PHPSTORM
    • Civi:
      • Server: civi
      • IDE Key: PHPSTORM
    • Smashpig:
      • Server: smashpit
      • IDE Key: PHPSTORM
  • In addition to the above, you would have to update the Xdebug web on payments and civicrm because there's a slight issue with docker on mac that affects Xdebug. Paste the following lines of code in <rootdir>/config/payments/xdebug-web.ini & <rootdir>/config/smashpig/xdebug-cli.ini xdebug.remote_autostart=on xdebug.remote_host=docker.for.mac.localhost
    xdebug.remote_autostart=on
xdebug.remote_host=docker.for.mac.localhost
    In <rootdir>/config/civicrm/xdebug-cli.ini
    xdebug.remote_autostart=off
xdebug.remote_host=docker.for.mac.localhost
    then restart docker’s smashpig and payments container to enable the change
  • Test payment-wiki
    1. add a breakpoint at /src/payments/extensions/DonationInterface/gateway_forms/Mustache.php setGateWay function
    2. Then select Payment-wiki as configuration
    3. click on the green bugs
    4. open localhost:9001 then click any payment links, you will see your xdebug is working
  • Test civi
    1. open phpstorm preferences -> php -> debug -> break at first line in php scripts
    2. add breakpoint at src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/org.wikimedia.smashpig/CRM/Core/Payment/SmashPigRecurringProcessor.php
    3. go to the civicrm docker container (./civicrm-drush.sh)
    4. run 
      docker@civicrm:/srv/civi-sites/wmff/drupal/sites/default/civicrm/extensions$ drush --user=1 -v -r /srv/civi-sites/wmff/drupal qc
      You might get the error saying that you should have the php_ide_config set up
    5. then inside the civicrm docker, run 
      export PHP_IDE_CONFIG="serverName=civi"
      the serverName will be the one you have set up at phpStorm server, the one have right path mapping
    6. skip the first break point, then it will work
  • Test smashpig
  1. set a breakpoint at src/civi-sites/wmff/vendor/wikimedia/smash-pig/Maintenance/ConsumePendingQueue.php execute function's new PendingQueueConsumer.
  2. inside the smashpig container, to go /srv/smashpig/Maintenance
    export PHP_IDE_CONFIG="serverName=smashpig"
    then 
    docker@smashpig:/srv/smashpig/Maintenance$ php ConsumePendingQueue.php

Day 9

Failmail

Understand the error messages, permission will be added by Jeff,

e.g. Check forget me processing failed with code 1

  • After check Acoustic > data -> contact list > view's each db with the email you saw from civi forget me, if no matches then just delete it.
  • Use this to can delete where id is equal to the most recent id as above xxxx
  • Go this page, hit execute, we should see a large number id in result.

Day 10

Deployment

  • Otherwise just ask your buddy to do that with you for the first time, and it’s always good to have someone else's recent deployment from Gerrit to use as ref.
  • After Jenkins gives you all the good feedback,  you can use this link
  1. Log in to the active fundraising deploy server frpm1001
    ssh frpm1001
  2. Run fundraising_code_update to refresh the deploy server's staging area
    fundraising_code_update -p civicrm (if payment use payment-wiki)
  3. Confirm the reported commit changes match what you expect, you can inspect the contents of the project staging directory which is listed by
    fundraising_code_update -i
  4. Run rsync_blaster to propagate this staged code to the individual servers
    rsync_blaster ALL:civicrm (if payment use payment-wiki)
  5. Log action into #wikimedia-fundraising by typing the below (where xyz & abc are the commits) into IRC !log Updating civicrm from xyz to abc (if payment use payment-wiki)
  6. Turn your stuff back on and make sure it works.

Note:

  • We do civicrm upgrade if only the core got upgraded, and all you need to know is this link
  • For the customized civicrm upgrade, we need to have the submodule update as well
  • For payment-wiki you just need to follow the deployment link above and replace the civicrm to payment-wiki
  • For fraud filter deployment, since it's private files, not in Gerrit, so just do directly to frpm1001. Nano the localSettings, update it and deploy it, once you are done, test a real world donation and see from the log if everything is going well.
  • Deploy to staging site: Fundraising/Cluster/Civicrm staging server

Day 11

EOY Email

(Usually Elliot will run the cron job, rest of the team will watch the whole FailMail):

To update these numbers (safely because this CiviCrm is the slave db):

  • Run `ssh frdev1001`
  • Type `mysql`
  • Run `use civicrm`

then

SELECT year, status, COUNT(*) FROM wmf_eoy_receipt_donor GROUP BY year , status;

To see them live (using prod mysql access: it's in the .my.cnf file in your home directory)

  • Run `ssh civi1001`
  • Change directory to `/srv/org.wikimedia.civicrm/drupal`
  • Run `drush civicrm-sql-cli`

then

SELECT year, status, COUNT(*) FROM wmf_eoy_receipt_donor GROUP BY year , status;

To preview an EOY email locally:

admin@admins-MBP ~ % cd ~/wiki
admin@admins-MBP wiki % ./civicrm-drush.sh

Render email to preview

docker@civicrm:/srv/civi-sites/wmff/drupal$ drush @wmff cvapi EOYEmail.Render version=4 limit=1 year=2022 contactID=210

Then 

docker@civicrm:/srv/civi-sites/wmff/drupal$ drush @wmff cvapi EOYEmail.Send version=4 limit=1 year=2022 contactID=210

Go then go to localhost:1080 (MailCatcher) we will see the test mail.

Day 12

FraudFilter

Refer to this Gerrit Ticket

Update the policy inside frpm1001:/localsettings/payment-wiki/LocalSettings.php, and do the deploy:

fundraising_code_update -p payment-wiki

then 

rsync_blaster ALL:payment-wiki

then go to the frlog1002:/var/log/remote to check the corespondent payment method log, then always try to make a real donation and check the log to make sure nothing is been break due to the new policy we add.

Since this is private file, which we do not have code publish to Gerrit, so there is no way to do the git review process, just ask you peer to do that with you if you are not very sure about the whole process.

Day 13

Emergency Protocol / Incident Response

Day 14

Email Preference

  • local test site: https://localhost:9002/index.php/Special:EmailPreferences/emailPreferences?contact_hash=2261039855&contact_id=6
  • local test with json response example: https://localhost:9005/rest.php?entity=civiproxy&action=getpreferences&key=SITE_KEY&version=3&json=1&api_key=API_KEY&hash=2261039855&contact_id=6
  • Queue consume files:
  1. /src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/wmf-civicrm/CRM/Queue/PreferencesQueueConsumer.php
  2. /src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/wmf-civicrm/api/v3/Preferencesqueue/Consume.php
  3. /src/civi-sites/wmff/drupal/sites/default/civicrm/extensions/wmf-civicrm/api/v3/Preferences/Create.php
  • Css file:
    • src/email-pref-ctr/extensions/DonationInterface/modules/css/emailPreferences.css
  • html file:
    • /src/email-pref-ctr/extensions/DonationInterface/email_forms/templates/emailPreferences.html.mustache
  • i18n file:
    • /src/email-pref-ctr/extensions/DonationInterface/email_forms/i18n/en.json
  • php file:
    • /src/email-pref-ctr/extensions/DonationInterface/special/EmailPreferences.php
  • Cmd to consume
    • drush cvapi Preferencesqueue.consume time_limit=1000 max_batch_size=1

Day 15

Monthly Convert

Currently we have Ingenico and Adyen those two methods to support the Monthly Convert, which we would love to use a Modal to ask our one time donor do they want to do the monthly donation after they one time payment.

Monthly convert Ingenico local test links:

  • https://localhost:9001/index.php?title=Special:IngenicoGateway&appeal=JimmyQuote&ffname=cc-vmad&payment_method=cc&recurring=0&uselang=en&language=en&currency=USD&amount=35&country=US&first_name=Jimmy&last_name=Wales&street_address=1+Montgomery+Street&city=San+Francisco&state_province=CA&postal_code=94104&email=jwales%40example.com&debug=true&variant=monthlyConvert
  • https://localhost:9001/index.php/Special:IngenicoGateway?debugMonthlyConvert=true

Set breakpoint @ src/payments/extensions/DonationInterface/gateway_common/RecurringConversion.api.php

Day 16

FR-tech-dev cloud project to set up amazon local env

Config the payments.fr-tech-dev.eqiad1.wikimedia.cloud for local sandbox amazon testing

First add below to .ssh/config

Host payments.fr-tech-dev

   User <yourname>

   Hostname %h.eqiad1.wikimedia.cloud

   ProxyJump bastion.wmcloud.org:22

   IdentityFile ~/.ssh/id_rsa

Then add the paired public key to this site:

https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack

Then run `ssh -vv payments.fr-tech-dev` to test if it is working, other wise need to contact your team to add you account as admin

then go the project root dir, run `./proxy-forward.sh`

Then you can use the paymentstest1.wmcloud.org as the payment site to do amazon testing now

P.s.

Amazon sandbox testing credential is:

nobody@ejegg.com / amazon

Day 17

How to watch log with logdog:

1: ssh to  civi1001 or frlog1002

2: copy this logdog.sh to your home dir

3: from your home dir, run: `bash logdog.sh -d <date> <keyword>` then all the qualified gz files will go to your home-directory/logdog/<keyword>

5: go to that directory contains all the keywords, then run something like `grep -r --color=always "fraud"` (which is equivalent to ‘grep --color=always "fraud" *’)then you will be able to see all other keywords for those files.


Or without logdog, you can also zgrep keyword:

xxx@frlog1002:/srv/archive/frlog1002/logs$ zgrep --color=always <keyword> *<date>.gz, to fetch all the gz files at date xxx see if they have the keyword xxx with keyword highlighted


p.s. can use `echo "<XML XXXXX>" | xmllint --format -` to view formated xml

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Fundraising/Team_processes/Onboarding&oldid=2146785"