VRT System/Upgrading

If there is sufficient time before the maintenance add it to next week's Tech News

VRT System
Configuration Customization Database Failover Installation Mail Setup SSL Troubleshooting Upgrading External resources: Production VRTS Cloud Test Installation Upstream VRTS Docs VRTS in Phabricator

Pre-requisites

• Let the users know of the upcoming downtime/maintenance through the following channels:
  • Notify volunteers on https://vrt-wiki.wikimedia.org on the Café page by creating a new topic. The title could be "VRTS Downtime" or "VRTS Maintenance".
  • Send an email to the otrs-admins mailing list.
  • Create a maintenance window on the VRTS dashboard: https://ticket.wikimedia.org/otrs/index.pl?Action=AdminSystemMaintenance
  • Post on the IRC #wikimedia-vrt channel (optional: not every volunteer is on IRC and the admins are already covered by the mailing list)

Automated Upgrade

You can upgrade the primary using the cookbook which is the recommended way at this point in time.

sudo cookbook sre.vrts.upgrade --help
usage: cookbook [GLOBAL_ARGS] sre.vrts.upgrade [-h] [-t TASK_ID] -r REASON --version VERSION host

Upgrade VRTS hosts to a new version

    Usage example:
        cookbook sre.vrts.ugprade --version 6.5.6 -r "Latest LTS Version" -t T12345 vrts1003
    

positional arguments:
  host                  Short hostname of the VRTS host to upgrade e.g. vrts1001

optional arguments:
  -h, --help            show this help message and exit
  -t TASK_ID, --task-id TASK_ID
                        The Phabricator task ID (e.g. T12345). (default: None)
  -r REASON, --reason REASON
                        Administrative Reason. The current username and originating host are automatically added. (default:
                        None)
  --version VERSION     Version of new VRTS installation (default: None)

Cookbook owner team: Collaboration Services

This will upgrade the host to the specified version which should be higher than the current version.

Post Upgrade

• Go to the VRTS package manager and re-install all packages: https://ticket.wikimedia.org/otrs/index.pl?Action=AdminPackageManager
• Check the vrts-daemon logs to see if everything works as expected:

sudo journalctl -fu vrts-daemon

• Upgrade the passive instance i.e. vrts2002 by running:

sudo /usr/bin/rsync --rsh /usr/local/sbin/sync-vrts-ssl-wrapper -av --progress rsync://vrts1003.eqiad.wmnet/vrts /opt/

• Test functionality by sending an mail to info-en@wikimedia.org and check that it shows up in info-en queue on the VRTS dashboard.

Manual Upgrade

This section is general guidance for a patchlevel (e.g. 6.5.x -> 6.5.y) update only. Minor or Major version upgrades can be complicated by database schema changes and other issues. There's really no way around reading the upgrade documentation, and testing the updates on a real system based on our existing configuration and database.

• Determine which server is currently the active server by looking at hieradata/role/common/vrts.yaml:profile::vrts::active_host in the puppet repo.
• Determine the currently installed version by looking at the target of the symlink /opt/otrs.
• Fetch new VRTS, extract into /opt and read through any upgrade instructions:

curl -L -O  https://download.znuny.org/releases/znuny-$VERSION.tar.gz
sudo tar xfz znuny-$VERSION.tar.gz -C /opt

• Stop the relevant services:

sudo disable-puppet "VRTS Upgrade"
sudo systemctl stop apache2
sudo systemctl stop exim4
sudo systemctl stop cron
sudo systemctl stop vrts-daemon

• Copy old config into new code tree & fix permissions:

sudo cp /opt/otrs/Kernel/Config.pm /opt/znuny-$VERSION/Kernel
sudo cp /opt/otrs/var/log/TicketCounter.log /opt/znuny-$VERSION/var/log/
sudo ./otrs/bin/otrs.SetPermissions.pl --otrs-user=otrs --web-group=www-data /opt/otrs

• Fix symbolic link, permissions and rebuild and install packages:

sudo ln -sfn /opt/znuny-$1 /opt/otrs
sudo -u root /opt/otrs/bin/otrs.SetPermissions.pl --web-group=www-data
sudo -u otrs /opt/otrs/bin/otrs.Console.pl Maint::Config::Rebuild
sudo -u otrs /opt/otrs/bin/otrs.Console.pl Maint::Cache::Delete

• Restart the relevant services:

sudo systemctl start cron
sudo systemctl start exim4
sudo systemctl start apache2
sudo systemctl start vrts-daemon
sudo enable-puppet "VRTS Upgrade"

• Run Puppet: sudo run-puppet-agent
• Refer to the Post Upgrade section above for actions to be performed after an upgrade.

This page is a part of the SRE Collaboration Services technical documentation
(go here for a list of all our pages)