Jump to content

Fundraising/techops/procedures/services-minio upgrade

From Wikitech

TLDR / YOLO / Speedrun

On puppetserver:

  • build_package minio

On all minio hosts:

  • sudo apt update && sudo apt upgrade

On one minio host:

  • mcli admin service restart mycloud --insecure

Prep

All steps require that you have a cluster defined in your mcli config. This is stored in your .mcli/config.json file. In all cases, we'll add the --insecure flag to mcli commands since we are using internal certificates.

IMPORTANT: minio needs to have the same version running on all hosts in a cluster. Make sure you have time and access to complete the full process before starting.

Verify the current server version of all hosts

We use the admin info to verify the current running server version. It also verifies that all hosts are up and healthy. 

(vb)dallas@franio2003:~$ mcli admin info mycloud --insecure
●  franio2001.frack.codfw.wmnet:9000
   Uptime: 1 minute 
   Version: 2024-10-02T17:50:41Z
   Network: 3/3 OK 
   Drives: 3/3 OK 
   Pool: 1

●  franio2002.frack.codfw.wmnet:9000
   Uptime: 1 minute 
   Version: 2024-10-02T17:50:41Z
   Network: 3/3 OK 
   Drives: 3/3 OK 
   Pool: 1

●  franio2003.frack.codfw.wmnet:9000
   Uptime: 3 minutes 
   Version: 2024-10-02T17:50:41Z
   Network: 3/3 OK 
   Drives: 3/3 OK 
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 7.1% (total: 1.3 GiB) │ 9                   │ 1            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

9 drives online, 0 drives offline, EC:4
(vb)dallas@franio2003:~$

Upgrade

Build new minio version

On the puppetserver, build the new minio version. 

(vb)dallas@frpm1002:~$ build_package minio

<-- snip out a bunch of release info -->

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
RELEASE.2025-05-24T17-08-30Z is latest available upstream
These are in bookworm-frack:
 - minio_20241002175041.0.0

Enter a release number or (q)uit (RELEASE.2025-05-24T17-08-30Z): RELEASE.2025-04-22T22-12-26Z

We are going to build the following:
package:  minio
release:  20250422221226 (RELEASE.2025-04-22T22-12-26Z)
version:  0.0
distro:   bookworm
maint:    Dallas Wisehaupt <dwisehaupt@wikimedia.org>
import:   /home/dallas/packages/minio/minio_20250422221226.0.0_amd64.deb

Build and import this package? (y/[n]): y
fetch_file: minio_20250422221226.0.0_amd64.deb already downloaded
fetch_file: minio_20250422221226.0.0_amd64.deb.sha256sum already downloaded
checksum_verify_file: minio_20250422221226.0.0_amd64.deb: OK
Add this package to bookworm-frack repository? (y/[n]): y
Deleting any unreferenced files from bookworm-frack
minio_20250422221226.0.0_amd64.deb: component guessed as 'main'
Exporting indices...
Successfully created '/var/www/reprepro/bookworm/dists/bookworm-frack/Release.gpg.new'
Successfully created '/var/www/reprepro/bookworm/dists/bookworm-frack/InRelease.new'
Deleting files no longer referenced...
deleting and forgetting pool/main/m/minio/minio_20241002175041.0.0_amd64.deb
Sync puppet servers? (y/[n])? y
------------------------------
Syncing frpm2002.frack.codfw.wmnet

rsync /etc/puppet

rsync /etc/puppet-private

rsync /var/lib/git

rsync /var/www/reprepro
*deleting   bookworm/pool/main/m/minio/minio_20241002175041.0.0_amd64.deb
.d..t...... bookworm/db/
>f..t...... bookworm/db/checksums.db
>f..t...... bookworm/db/packages.db
>f..t...... bookworm/db/references.db
>f..t...... bookworm/db/release.caches.db
>f..t...... bookworm/db/version
.d..t...... bookworm/dists/bookworm-frack/
>f..t...... bookworm/dists/bookworm-frack/InRelease
>f..t...... bookworm/dists/bookworm-frack/Release
>f..t...... bookworm/dists/bookworm-frack/Release.gpg
.d..t...... bookworm/dists/bookworm-frack/main/binary-amd64/
>f.st...... bookworm/dists/bookworm-frack/main/binary-amd64/Packages
>f.st...... bookworm/dists/bookworm-frack/main/binary-amd64/Packages.gz
>f.st...... bookworm/logs/log
.d..t...... bookworm/pool/main/m/minio/
>f+++++++++ bookworm/pool/main/m/minio/minio_20250422221226.0.0_amd64.deb
(vb)dallas@frpm1002:~$ build_package mcli

Once you have the new version built, you can install it on all the hosts.

Install the new version on all hosts

In most cases you will want to force the update to ensure all hosts are on the same version in case of unexpected host/service restarts.

This can be done with sudo apt update && sudo apt upgrade. If you wish to just install the one package (in case there are other updates queued you don't want to apply yet. You can use sudo apt install --only-upgrade minio

This is an example of the install of just the minio package with additional updates queued: 

(vb)dallas@franio2003:~$ sudo apt update
[sudo] password for dallas: 
Hit:1 http://debian.csail.mit.edu/debian bookworm-updates InRelease
Get:2 http://frpm2002.frack.codfw.wmnet/reprepro/bookworm bookworm-frack InRelease [14.7 kB]           
Hit:3 http://security.debian.org/debian-security bookworm-security InRelease                           
Hit:4 http://debian.csail.mit.edu/debian bookworm InRelease                                            
Get:5 http://frpm2002.frack.codfw.wmnet/reprepro/bookworm bookworm-frack/main amd64 Packages [5,225 B] 
Fetched 19.9 kB in 0s (70.6 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
9 packages can be upgraded. Run 'apt list --upgradable' to see them.
(vb)dallas@franio2003:~$ sudo apt install --only-upgrade minio
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libgnutls-dane0 libidn12 libiw30 libnl-route-3-200 libunbound8
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
  minio
1 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
Need to get 41.3 MB of archives.
After this operation, 14.8 MB of additional disk space will be used.
Get:1 http://frpm2002.frack.codfw.wmnet/reprepro/bookworm bookworm-frack/main amd64 minio amd64 20250422221226.0.0 [41.3 MB]
Fetched 41.3 MB in 4s (11.0 MB/s) 
Reading changelogs... Done
(Reading database ... 50343 files and directories currently installed.)
Preparing to unpack .../minio_20250422221226.0.0_amd64.deb ...
Unpacking minio (20250422221226.0.0) over (20241002175041.0.0) ...
Setting up minio (20250422221226.0.0) ...
(vb)dallas@franio2003:~$

Restart the minio service

Once the software is installed across all hosts, we need to coordinate the restart across all instances. This can be done using the mcli command. 

(vb)dallas@franio2003:~$ mcli admin service restart mycloud --insecure
Service status: ▰▰▱ [DONE]
Summary:
    ┌───────────────┬─────────────────────────────┐
    │ Servers:      │ 3 online, 0 offline, 0 hung │
    │ Restart Time: │ 11.862729ms                 │
    └───────────────┴─────────────────────────────┘
(vb)dallas@franio2003:~$

Post update verification

You can verify the servers are running the new version using the mcli admin info command again: 

(vb)dallas@franio2003:~$ mcli admin info mycloud --insecure
●  franio2001.frack.codfw.wmnet:9000
   Uptime: 23 minutes 
   Version: 2025-04-22T22:12:26Z
   Network: 3/3 OK 
   Drives: 3/3 OK 
   Pool: 1

●  franio2002.frack.codfw.wmnet:9000
   Uptime: 23 minutes 
   Version: 2025-04-22T22:12:26Z
   Network: 3/3 OK 
   Drives: 3/3 OK 
   Pool: 1

●  franio2003.frack.codfw.wmnet:9000
   Uptime: 23 minutes 
   Version: 2025-04-22T22:12:26Z
   Network: 3/3 OK 
   Drives: 3/3 OK 
   Pool: 1

┌──────┬───────────────────────┬─────────────────────┬──────────────┐
│ Pool │ Drives Usage          │ Erasure stripe size │ Erasure sets │
│ 1st  │ 7.1% (total: 1.3 GiB) │ 9                   │ 1            │
└──────┴───────────────────────┴─────────────────────┴──────────────┘

13 B Used, 1 Bucket, 2 Objects
9 drives online, 0 drives offline, EC:4
(vb)dallas@franio2003:~$

You can also test listing, cat'ing, adding, or deleting files from any of the nodes:

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Fundraising/techops/procedures/services-minio_upgrade&oldid=2306981"