Fundraising-dev is a set of docker images coordinated by docker-compose configuration and a setup script. Its purpose is to let all members of the Fundraising Tech team develop against the same setup of PHP, mediawiki, and CiviCRM, along with their ancillary tools and databases. It also aims to speed team onboarding by making the install process as quick and painless as possible.

More Docs

There are more docs on the projects readme, write this blurb better:

https://github.com/wikimedia/wikimedia-fundraising-dev/

Quickstart

Clone the repo from https://gerrit.wikimedia.org/r/admin/repos/wikimedia/fundraising/dev,general
Run setup.sh

Detailed Start

Things you need before starting: 1. Docker installed 2. Your SSH key added to gerrit 3. A yubikey set up and access to frpm

1. clone above

2. Install docker https://docs.docker.com/engine/install/ubuntu/

3. Install docker-compose https://docs.docker.com/compose/install/

4. If you get an error docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', PermissionError(13, 'Permission denied'))

Add your user to the docker group

sudo gpasswd -a $USER docker

newgrp docker https://stackoverflow.com/questions/64662372/docker-compose-up-error-while-fetching-server-api-version-connection-aborte

(remember to have your user name as the file and group owner for the whole folder as sudo chown -R USER:GROUP *)

4.5 Add your SSH key to gerrit https://www.mediawiki.org/wiki/Gerrit/Tutorial#Set_Up_SSH_Keys_in_Gerrit If this isn't added ./setup.sh will fail with permission errors:

5. If the gitclone steps fail because of a permission error fatal: Could not read from remote repository.

Please make sure you have the correct access rights and the repository exists.

Components installed

CiviCRM

Two sites are created.

'wmff' is meant to resemble as close as possible the CiviCRM instance installed in WMF production, with all of our custom Civi extensions and drupal modules.

'dmaster' is built directly on core CiviCRM code and is meant to facilitate sending patches upstream.

If your setup.sh script fails building Civi with an error like this: "Error: Class 'Civi\Api4\Afform' not found in afform_civicrm_alterMenu() (line 396 of /srv/civi-sites/wmff/drupal/sites/all/modules/civicrm/ext/afform/core/afform.php)."

You can try clearing caches this way to get it working

./memcached-purge.sh && ./queues-redis-clear.sh

Payments-wiki

The fundraising branch of MediaWiki core, with all the same extensions that are deployed to payments.wikimedia.org, and a similar configuration. The default appeal from production (text shown next to the donation form) is imported during setup to make forms look as similar to production as possible. The main page has handy links to payment forms for various combinations of country, currency, and payment method.

Email Preference Center

Another MediaWiki instance on the same branch as payments-wiki but with different configuration. Payment processor forms are not enabled, but email preference forms are, and networking is configured to allow tightly controlled access to some CiviCRM data.

CiviProxy

An API proxy that allows tightly controlled access to CiviCRM data. We use it to mediate between the Email Preference Center and the CiviCRM database so that only one specific API request is allowed, and that the request's parameters are validated before being forwarded to CiviCRM. (CiviProxy Github site)

IPN listeners

A standalone SmashPig install, with Apache configured to send requests to the smashpig_http_handler endpoint. In payment processor sandbox configuration pages you can request IPNs be sent to e.g. https://paymentsipntest1.wmcloud.org/adyen (see FR-tech-dev cloud project, below).

PrivateBin

Used to comply with donor data requests. We have a donor-facing URL that only allows downloading and an internal-facing URL that allows uploading. (PrivateBin project site)

Mailcatcher

A development tool to catch any outbound mail sent by the other services and show it in a web UI.

Redis

Used as a queue server to allow communication between other components, as in production

MariaDB

Database server, as in production

Running stuff on the docker VMs

TODO

FR-tech-dev cloud project

This is a Wikimedia Cloud Services project with a single VM that primarily serves to allow access to our development machines via routable URLs (similar to the commercial service NGrok). Payment methods such as Apple Pay and Amazon Pay require forms to be served on URLs that can be crawled by the controlling entities. For any payment processor, testing IPNs requires a routable URL.

The setup.sh script has you select a 'proxy forwarding ID' - this ID determines which URLs and ports are reserved for you. To connect the routable URLs to your local machine, run

./proxy-forward.sh

in your local fundraising-dev root directory.

Domain Names and forwarding

The VM has twelve domain names pointing to it via the WMCS https web proxy. paymentstest1.wmcloud.org through paymentstest6.wmcloud.org are for payments forms, and paymentsipntest1.wmcloud.org through paymentsipntest6.wmcloud.org are for IPN testing.

The IPN test domains are configured to point directly to ports that are served from developer machines. The paymentstest domains are served via nginx, which serves the .well-known directory from the VM itself and forwards any other requests to developer machines. This is necessary so that the .well-known files are available any time that Apple crawls the site to verify our ownership of the domains.

Setup and Administration

The project is configured via Horizon. Log in with your Wikitech credentials and select the 'fr-tech-dev' project from the dropdown at top left. If not available, ask another fr-tech member to add you to the project.

To allow port forwarding, gateway_ports is enabled in ssh server settings. In Horizon, under Compute->instances->'payments'->Puppet Configuration, the Heira Config includes

profile::ssh::server::gateway_ports: true

Proxied domain names are configured under DNS->Web Proxies. paymentstest1-6 proxy to ports 8051-8056 on the payments VM, and paymentsipntest1-6 proxy to ports 8101-8106 on the payments VM.

Apple Pay domain verification files are stored in /var/www/applepay/paymentstest[1-6]/.well-known directories.

Ngnix configuration to serve .well-known locally and forward other requests is as follows (example for paymentstest1):

server {
        listen 8051;
        server_name paymentstest1.wmcloud.org;
        include custom_502;

        location /.well-known {
                alias /var/www/applepay/paymentstest1/.well-known;
        }

        location / {
                proxy_set_header Host $host;
                proxy_pass http://localhost:8001;
        }
}

The custom_502 config file serves a default page when no development machine is answering on the proxied port:

error_page 502 =200 /error-502.html;

location /error-502.html {
	internal;
	root /var/www/fallback;
}

Images

Dockerfiles for the necessary images are checked into the Release Engineering team's dev-images repo, hosted on the WMF gitlab instance. After you get a change merged into the dev-images repo, you may have to ask a member of release engineering to compile the image and upload it to the WMF docker registry.

fundraising-buster
fundraising-buster-php73
fundraising-civicrm-buster-php73
fundraising-civicrm-buster-php73-apache2
fundraising-civiproxy-buster-php73-apache2
fundraising-mediawiki-buster-php73-apache2
fundraising-privatebin-buster-php73-apache2
fundraising-smashpig-buster-php73-apache3