Metrics Platform/How to/Deploy a New Client Library Version

Java

The Java client library is published to Maven Central with an auto-mirror at WMF Archiva (note that only when versions are requested do the release versions get auto-mirrored).

To publish a new release, artifacts must be uploaded to Sonatype OSS (Maven Central) which requires the proper permissions of the target repository to push and create signed/annotated tags, and to sign the artifacts.

Currently publishing a new release is a manual process (there is a ticket in our backlog to automate this on demand) whereby an engineer needs to store encrypted credentials in order to upload artifacts. See the permissions section of the parent pom repository for instructions on how to store credentials, setup a GPG key, etc.

Once an engineer has all the requisite, valid credentials, the following commands can be run from the root Metrics Platform directory to publish a new release:

./mvnw -f java -B -P deploy-central release:clean
./mvnw -f java -B -P deploy-central release:prepare
./mvnw -f java -B -P deploy-central release:perform

Versioning

Running through the release steps above will result in a minor version bump of the Java client library automatically. If a major version bump is needed, a merge request should be submitted to the Metrics Platform repository with an update to the major version number:

    <groupId>org.wikimedia.metrics</groupId>
    <artifactId>metrics-platform</artifactId>
    <version>3.0-SNAPSHOT</version>

Here is an example merge request for releasing version 2.0 of the Java client library from 1.14: [Java] Bump snapshot version, remove tag from pom file.

JavaScript

The JavaScript client library is embedded in the EventLogging MediaWiki extension (EventLogging), which means that it is deployed to the Wikimedia Foundation servers weekly through the Deployment Train. With deployment taken care of for us, we can focus on updating the client library embedded in EventLogging.

We use the maintenance/manageForeignResources.php maintenance script (the maintenance script) to manage third-party front-end assets in MediaWiki Core or an extension. The script reads a foreign-resources.yaml file, which lists the assets that should be fetched, where they should be fetched to, and how they can be verified. For MediaWiki Core, the file is located in the resources/lib/ directory; for EventLogging, the file is located in the modules/lib/ directory.

Therefore, to update the client library, we:

1. Update the EventLogging/modules/lib/foreign-resources.yaml file
2. Use the maintenance script to fetch and verify the client library
3. Submit a patch for review

Update the EventLogging/modules/lib/foreign-resources.yaml file

First update the commit hashes:

metrics-platform:
  license: GPL-2.0+
  homepage: https://wikitech.wikimedia.org/wiki/Metrics_Platform
  version: $SHORT_COMMIT_HASH
  type: tar
  src: https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/archive/$COMMIT_HASH/metrics-platform-$COMMIT_HASH.tar.gz
  integrity: sha384-$INTEGRITY
  # GitLab nests their archives with a variable subdirectory...
  dest:
    metrics-platform-*/COPYING:
    metrics-platform-*/js/src/ContextController.js:
    metrics-platform-*/js/src/ContextUtils.js:
    metrics-platform-*/js/src/CurationController.js:
    metrics-platform-*/js/src/MetricsClient.js:
    metrics-platform-*/js/src/SamplingController.js:
    metrics-platform-*/js/src/StreamConfigUtils.js:

Next, ensure that the dest section is up to date, adding or removing entries where necessary. Finally, generate a value for the integrity property using the maintenance script:

$ php maintenance/run.php ./maintenance/manageForeignResources.php --extension EventLogging make-sri
... checking 'metrics-platform'
Integrity for https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/archive/2682156f07eb12b6366597f28819bfd8d05b46c7/metrics-platform-2682156f07eb12b6366597f28819bfd8d05b46c7.tar.gz
	integrity: sha384-lXXKT8SBrnRfXDffHGh90THwB1xiwMifpEqH7gGjn8y3sh1mLU19cCLjOdvLa7Ub

Fetch and verify the client library

Fetch and verify the client library using the maintenance script:

$ php maintenance/run.php ./maintenance/manageForeignResources.php --extension EventLogging update
$ php maintenance/run.php ./maintenance/manageForeignResources.php --extension EventLogging verify
... verifying 'metrics-platform'

Submitting a patch for review

By convention, we list the subject lines of all commits to the client library since the previous version. For example, the following is the full message for a commit that updates the client library to 1f3813bd9b86:

lib: Update lib/metrics-platform to 1f3813bd9b86
  
  $ cd /path/to/metrics-platform
  $ git log --pretty=format:%s 20ae4f2e566a..1f3813bd9b86 js
    
    [PHP][JS][Java] Update schema paths in clients, tests.
    [JS][PHP] Update DocBlocks
    [JS] Document new interfaces
    [JS] ElementInteraction -> ElementInteractionData
    [JS] Update click schema ID
    [JS] Update MetricsClient#submitInteraction() signature
    [JS] Support adding context values based on stream config
    [JS] event -> eventData in MetricsClient#submitInteraction()
    [JS] Fix incorrect schema title
    [JS] Add integration test for MetricsClient#submitInteraction()
    [JS] Test happy path of MetricsClient#submitInteraction()
    [JS] Interaction -> InteractionData
    [JS] Add schemaID param to MetricsClient#submitInteraction()
    [JS] Update docs for MetricsClient#submitInteraction()
    [JS] Update ESLint rules
    [JS] Add MetricsClient#submitInteraction()
    [JS][Java] Mix in sample unit and rate
    
Supporting changes:
    
* Define mw.eventLog.submitInteraction() and submitClick()

PHP

Like the JavaScript client library, the PHP client library is used in the EventLogging MediaWiki extension (EventLogging), which means that it is deployed to the Wikimedia Foundation servers weekly through the Deployment Train. However, the PHP client library is not embedded in EventLogging. Like all other PHP libraries, it is managed using Composer and, like all other required or recommended libraries deployed to the Wikimedia Foundation servers, it is managed in the mediawiki/vendor repository. Confusingly though, it is also managed in EventLogging as, in principle, MediaWiki extensions can be hacked on in isolation.

Therefore, to update the client library we need to use Composer to manage the library in EventLogging and the mediawiki/vendor repository:

cd /path/to/mediawiki/extensions/EventLogging
composer self-update 2.6.4
composer require wikimedia/metrics-platform $VERSION
composer update --no-dev

cd /path/to/vendor
composer self-update 2.6.4
composer require wikimedia/metrics-platform $VERSION
composer update --no-dev

See also the detailed instructions in the README in the mediawiki/vendor repository.