Juniper switch upgrade
Download the JTAC Recommended Software
First you need to download the JTAC Recommended Junos Software for the EX4300 at the link below.
Note: You need to have a Juniper account and privilege to download software. If you try to download the software and you get a message saying that you need privilege, just open a SR with Juniper and request access to download software.
https://kb.juniper.net/InfoCenter/index?page=content&id=KB21476
Once you click on the link above in the EX4300 session at the moment of the tutorial the recommended version is Junos 18.1R3-S6. Click on EX4300 and this will open another page on that page, (link below)
https://support.juniper.net/support/downloads/
under version, select 18.1 then download the one with “EX4300 Series 18.1R3” to your laptop.
After accepting the License agreement, select the option “To download the image on your localhost, CLICK HERE”
In the download column, click on "checkSums" the MD5 for the 18.1R3 image at the moment of this tutorial is
MD5 : d6aef35be1a76d729c134d926c6f327a
Navigate where you saved the image file to do a checksums of the image file.
papaul@papaulpc:~/Downloads$ md5sum jinstall-ex-4300-18.1R3.3-signed.tgz d6aef35be1a76d729c134d926c6f327a jinstall-ex-4300-18.1R3.3-signed.tgz
We can see that the MD5 we have on Juniper web site is the same as the MD5 on our laptop. Copy the the file from your computer /laptop to a USB
Known issues
- when upgrading from pre-21.2R1 to 21.2R1 or later,
no-validate
is required
Connect the new EX4300
Rack/power the new EX4300 in any rack and connect just the console cable to it (no network cable). Update Netbox with the new switch information. As temporary name we can use msw2-eqiad
login to the new EX4300 using console (default user name root, default password : leave empty)
Note: Once you login, you will see that the EX4300 will be trying to connect to a DHCP server to do upgrade and it will be annoying . To turn this off, you need to set a temporary root password and issue the command “Delete chassis auto-image-upgrade”
set system root-authentication plain-text-password
enter the temporary password twice and "commit"
root> delete chassis auto-image-upgrade root> commit
Copy the file from your USB to the new EX4300
Identify your USB drive
Before you connect the USB to the switch, issue the command
root@% ls /dev/da*
You will see an output similar or close to below
/dev/da0 /dev/da0s1c /dev/da0s2a /dev/da0s3 /dev/da0s3e /dev/da0s1 /dev/da0s1f /dev/da0s2c /dev/da0s3c /dev/da0s1a /dev/da0s2 /dev/da0s2f /dev/da0s3d
connect the USB and issue the same command you will likely see that new devices have been added
root@% ls /dev/da* /dev/da0 /dev/da0s1c /dev/da0s2a /dev/da0s3 /dev/da0s3e /dev/da0s1 /dev/da0s1f /dev/da0s2c /dev/da0s3c /dev/da1 /dev/da0s1a /dev/da0s2 /dev/da0s2f /dev/da0s3d /dev/da1s1
In the example above, you'll see both "/dev/da1" and "/dev/da1s1" have been added for the USB drive. When we mount the USB drive, we will be using the device with "s1" at the end of the name.
Mount the USB drive
Create a directory for the USB drive to mount to:
root@% mkdir /var/tmp/usb
Mount the USB drive to the /var/tmp/usb directory:
root@% mount_msdosfs /dev/da1s1 /var/tmp/usb root@% ls /var/tmp/usb jinstall-ex-4300-18.1R3.3-signed.tgz
Note: 'ls /var/tmp/usb' will show all the files on the USB drive.
Copy the Juniper file from the USB to the /var/tmp/usb directory:
root@% cp /var/tmp/usb/jinstall-ex-4300-18.1R3.3-signed.tgz /var/tmp root@% ls /var/tmp jinstall-ex-4300-18.1R3.3-signed.tgz
The jinstall-ex-4300-18.1R3.3-signed.tgz file from the USB is successfully copied to the /var/tmp directory.
Note: At this point, you can do another checksum to make sure that the file copied to the switch was not compromised.
root@% file checksum md5 /var/tmp/jinstall-ex-4300-18.1R3.3-signed.tgz
Now unmount the USB drive
root@% umount /var/tmp/usb
Start the upgrade
The two commands you need to run the upgrade on the switch from the CLI are:
root> request system software add /var/tmp/jinstall-ex-4300-18.1R3.3-signed.tgz root> request system reboot
Verification
After the reboot, you can issue the command "show version" to check the version running on the swith
root> show version Junos: 18.1R3.3
For EX switches, ensure that the primary and backup partitions are running the same version:
root> show system snapshot media internal
root> request system snapshot slice alternate
root> show system snapshot media internal
Copy old switch configuration to new switch
Open 2 terminal windows, in the first terminal window open the old switch and in the second terminal open the new switch.
on the old switch window issue the command
root> show configuration
copy the configuration and paste it in the new switch terminal.
Important: Do not copy all the users and their SSH keys because over console this will be very very slow. What you need to do is to just copy first all the configuration without the users ssh keys then copy two or 3 users with their SSH keys one by one. You can add the other users and their SSH keys after you login to the switch using ssh and not console.
Note: Do not copy the root password also.
on the new switch issue the command "show configuration" and double check that the configuration on the new switch matches the configuration on the old switch then commit
Virtual Chassis switches
The following change is needed before any upgrade otherwise the upgrade process fails to copy the image to the other VC member with rcmd: connection timeout
[edit system]
- internet-options {
- tcp-drop-synfin-set;
- no-tcp-reset drop-all-tcp;
- }
Use the force-host keyword to make sure the switch base OS is upgraded as well: request system software add <junos-image> force-host
Monitoring downtime
- Icinga an AlertManager :
sudo cookbook sre.hosts.downtime --hours 2 -r "Switch upgrade" -t XXXX 'P{P:netbox::host%location ~ "LOCATION"}'
- where LOCATION is for example "eqsin" or "A.*eqiad"
- Additional services might need to be downtimed manually (eg. LVS VIPs)
- AlertManager: in the https://alerts.wikimedia.org/ dashboard, create a new Silence with "site=eqsin" and "severity=page" (for example)
- The list of alerts for a given site is available on https://prometheus-eqsin.wikimedia.org/ops/classic/alerts (change the FQDN to match the wanted site).
- AlertManager: in the https://alerts.wikimedia.org/ dashboard, create a new Silence with "site=eqsin" and "severity=page" (for example)
- LibreNMS scheduled maintenance : https://librenms.wikimedia.org/alert-schedule