Out-of-band network
All of our sites have a dedicated Out Of Band (OOB) network, consisting of internet access (DIA), routers and switches physically separated from our production network.
This OOB access is our last resort solution in case of a site becoming partially (eg. bastions) or fully (eg. transport/transits) unreachable by normal means.
Note that in the event of a site losing connectivity to the internet, it's still reachable through its transport links from any other bastion host.
Prerequisites
To use our OOB network you need to have:
- An account on the network devices, see the
userslist in common.yaml - The password store checked out on your local machine
Step by step
The entry points to SSH to our management routers over their DIA are:
| site | hostname |
|---|---|
| eqiad | mr1-eqiad.oob.wikimedia.org |
| codfw | mr1-codfw.oob.wikimedia.org |
| esams | mr1-esams.oob.wikimedia.org |
| ulsfo | mr1-ulsfo.oob.wikimedia.org |
| eqsin | mr1-eqsin.oob.wikimedia.org |
| drmrs | mr1-drmrs.oob.wikimedia.org |
| magru | mr1-magru.oob.wikimedia.org |
NOTE: You should make sure that your SSH config does not have any entries that will try to connect to the above hostnames via one of our bastion hosts. To ensure your connection cannot be affected by any in-band changes to the network you need to ensure it goes directly from your machine to the OOB link over the internet.
From there you can SSH directly to relevant devices on the OOB network, for example a console server, eg:
mr1-eqiad> ssh root@scs-a8-eqiad.mgmt.eqiad.wmnet
Where you will need the matching password from the password store.
Note that it's also possible do to agent forwarding and port forwarding on the management routers to reach devices with password authentication turned off.
SSH Algorithms
On some of the more recent JunOS versions you may find the management router cannot connect to older devices, due to the support algorithms/ciphers supported on the destination device not being allowed by default:
cmooney@mr1-eqiad> ssh root@scs-f8-eqiad.mgmt.eqiad.wmnet Unable to negotiate with 10.65.0.4 port 22: no matching MAC found. Their offer: umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com
To work around this you can drop to a csh shell and use the regular ssh client, specifying the correct parameters manually:
cmooney@mr1-eqiad> start shell % % ssh -m hmac-sha2-512-etm@openssh.com root@scs-f8-eqiad.mgmt.eqiad.wmnet Password: #
No DNS
If DNS resolution doesn't work (for internal or external hosts), you can replace the FQDNs with the IPs listed in our Puppet configuration (OOB, scs, routers, etc)
To protect against the eventuality of being unable to resolve wikitech-static.wikimedia.org to reach this page, you can save a copy of this page locally or look it up in the Wayback Machine. In both cases, there is of course a risk that the last saved version is outdated; also, you would have to know about these workarounds on your own, being unable to look them up here on Wikitech.
See also
- tunnelencabulator - alternate-PoP IP address ssh tunnel management tool