Analytics/Systems/Cluster/Deploy/Refinery-source

From Wikitech
Jump to navigation Jump to search

Refinery-source is the JVM software on the Analytics Cluster using spark or hive UDFs. The source code is in the analytics/refinery/source.

How to deploy with Jenkins (and related steps)

Before starting, please check:

  • Check the latest refinery-source deployed version. This can be done in various ways on the git repository, for instance:
  • The Refinery source commit list and make sure that the changelog.md has been updated with the latest version and possibly that [maven-release-plugin] has committed the related version bump changes (the last step is optional, it could be triggered manually following the instruction below).
  • Make sure you are logged in to Jenkins to see the pages linked in the steps below.

Have you read the instructions reported above? If so, please keep going, otherwise please read them!

Deploy procedure:

  1. Update the changelog.md file at the root of the repository with changes that are going to be deployed - commit and merge this change.
  2. Release a new version of refinery-source jars to Maven: Visit https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/build?delay=0sec and click the "Build" button (default parameters shouldn't be changed except if you know what you're doing :). The job should take a few minutes to run, and an email is being sent to the analytics-alert list with whether the job succeeded or failed (it is also indicated on the page used to launch the job if you don't close it).
  3. NOTE: it's best to wait a few minutes between the build and this step. Now we need to update the symlinks to the latest refinery-source jars in the refinery repository. This can also be done via Jenkins, by going to https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/build, and setting the RELEASE_VERSION field to the version number of the latest jars released, without the v prefix. Example: v0.0.40 is bad, 0.0.40 is good. Then hit Build. This will send a code-review to gerrit in the refinery repository and send a message to the #wikimedia-analytics IRC channel as well as an email to the analytics-alerts mailing list. Look at the code review generated by Jenkins, it should contain the new jars and updates for the default-jars symlinks. If it looks good (normally, yes :), you can merge it and continue the deployment process. NOTE: This is new, make sure you merge!
  4. There are most chances that, having deployed some new/changed java/scala code, you wish to apply it in refinery oozie job(s). You then need to

If the maven release job failed (step 2)

First find why looking at job logs, and then skip the version number you were currently deploying

    • Update the changelog,md with the skipped version info
    • Deploy again!

Administration

Changing the archiva-ci password

Archiva expires user passwords periodically (every 90 days?). This means we need to update the archiva-ci user's password in Archiva, and also in the Jenkins stored credential.

Log into archiva.wikimedia.org as the admin user (the password is in the ops pwstore). Go to 'Manage', click on the edit (pencil) button for the archiva-ci and change the password.

Then, log into jenkins and go to the archiva-ci credentials page. Click on the 'Update' link, then click on Change Password, and set the password to the same one you set in Archiva.

How to deploy from the CLI

  • Update the changelog.md file at the root of the repository whith changes that are going to be deployed.
  • Prepare deployment (change pom.xml files, push to git):
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:prepare
  • Check everything looks Ok:
cat release.properties
  • Actually deploy (jar generation and uploads to archiva)
mvn -Duser.name=YOUR_WIKITECH_USERNAME release:perform

Please see the refinery page to deploy the jars and oozie code.