Production shell access

From Wikitech
(Redirected from Requesting shell access)
Jump to: navigation, search
Shells!

This page deals with shell access to the production Wikimedia cluster. For shell access to Labs, see Help:Access.

The first rule to keep in mind here is that production access is extremely sensitive. Treat it that way by closely following the rules in the server access responsibilities document.

If you have questions about security or accidentally break these rules, contact Tech Ops.

Requesting access

Production shell access is granted strictly on an as needed basis, and is entirely under the purview of the Engineering Department and Operations Department managers. They can approve or deny access for any reason, as security is of the highest priority.

To acquire shell access, you have projects or responsibilities that requires this access on a regular and ongoing basis. Requests based on a one-time need will not be granted. If you have a one-time need for data, request the data instead.

You will need a Phabricator account to complete either of these processes. If you don't have one, see the instructions here.

New users

  1. Read, comprehend, and sign the Acknowledgement of Wikimedia Server Access Responsibilities.
  2. Sign up for developer access if you haven't already.
  3. Use this form to create a ticket requesting access.[1]
  4. In the title, replace "RESOURCE" and "USER" with your name and the resource you need access to. (For new user requests, make a separate ticket for each user.)
  5. Add the following information to the description:
    • Your full name
    • Your developer access username (that is, the one you use for labs SSH, not Wikitech login. Wikitech shows this as "instance shell account name" in preferences). We will use this as your production shell username.
    • The public key from your SSH keypair.[2] This must not be the same one you use to access Labs.
    • A detailed reason for your request. In particular, describe which specific servers you need access to and why. We err on the side of giving fewer permissions rather than more, so the more detailed your request, the more likely you are to get all the permissions you need.
  6. Get approvals from the following people. These approvals should be as comments to the Phabricator task, so these people will need a Phabricator account as well. The comments should be made directly through the web interface, not via email.[3]
    • At least one comment of support from a Wikimedia Foundation employee, explaining why it is a good idea to accept your request. The comment of support should be from your supervisor if you're an employee, or from the employee you will be collaborating with if you're not.
    • The project lead where your access will be granted.
  7. For most requests, a three business day waiting period must be observed after the request is filed.[4]
  8. When your request is approved, you will be asked to provide your full legal name, preferred email address for contact, and physical address to the Wikimedia Foundation Legal Team (or your employee contact may forward this information on your behalf). This information will be used to customize a non-disclosure agreement, which you will be asked to read, comprehend, and electrically sign through the Foundation's contract management system. The agreement will be similar to the Volunteer NDA.
  9. The Wikimedia Foundation employee that will be supervising your work will coordinate final sign off by a C-level staff of the Wikimedia Foundation when all other criteria have been met before your access is granted.

If you feel an unreasonable amount of time has passed, you can comment on the ticket to request update and/or request an update directly from the Operations team member on Ops Clinic Duty that week.

Additional permissions for existing users

To escalate shell access, you should be working on a project that requires this access on a regular and ongoing basis. Any one time requests should simply request what data is required, access is not granted for one time requests.

  1. As the Phabricator document to sign/acknowledge is new (as of January 2015), any escalations of existing users should include said user reviewing and acknowledging the Acknowledgement of Wikimedia Server Access Responsibilities document.
  2. Use this form to create a ticket requesting access.[1]
  3. In the title, replace "RESOURCE" and "USER" with your name and the resource you need access to. (Group tickets are acceptable when a group is being escalated.)
  4. Add the following information to the description.[5]
    1. Your full name
    2. Your shell username
    3. A detailed reason for your request. In particular, describe which specific servers you need access to and why. We err on the side of giving fewer permissions rather than more, so the more detailed your request, the more likely you are to get all the permissions you need.
  5. A three business day waiting period must be observed after the request is filed.[4]
    • This may not be required when the change is correcting a previous request, but should be followed for escalations that include not previously approved permissions. It may not be required in some other circumstances.

Technical details

Production shell users, their keys, and their permissions are managed in modules/admin/data/data.yaml in the operations-puppet repository.

SSH configuration

Security

Do not use SSH agent forwarding (the -A command line option). Agent forwarding does not make it possible to steal your key itself, but it does make it possible for someone to hijack your SSH agent and thus your identity, so we do not do it. The -a option (with a lower case "a") disables agent forwarding, and is thus included in the sample configurations below.

You should also add the following to your SSH config to ensure that you are not vulnerable to a serious SSH bug that was disclosed in January 2016:

Host *
    UseRoaming no

Newer SSH versions may have fixed this bug, in which case you will need to remove these lines from your config to connect.

Standard config

The standard configuration (people not having root access), is to have the ssh connection to be established on a bastion and proxy the command to the target host inside the cluster. To do this, add the following to your SSH config file (usually located at ~/.ssh/config).
Host bast1001.wikimedia.org
    # Direct connection for the bastion host
    ProxyCommand none
    ControlMaster auto

Host *.wikimedia.org *.wmnet !gerrit.wikimedia.org !git-ssh.wikimedia.org
    User your_username_here
    # Everything else goes via bastion acting as a proxy
    ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org
    # Do not offer other identities loaded in ssh-agent
    IdentitiesOnly yes
    IdentityFile ~/.ssh/your_production_ssh_key

In the example above you may replace bast1001.wikimedia.org with the bastion that is physically closest to you:

  • bast1001.wikimedia.org in the eqiad cluster in Virginia, United States
  • bast2001.wikimedia.org in the codfw cluster in Texas, United States
  • bast3002.wikimedia.org in the esams cluster in Amsterdam, The Netherlands
  • bast4001.wikimedia.org in the ulsfo cluster in San Francisco, United States
Map of bastion hosts

A map of the world showing where Wikimedia data centers are located.

Ops config

## Production & External Zones
Host iron.wikimedia.org bast1001.wikimedia.org bast2001.wikimedia.org bast3002.wikimedia.org bast4001.wikimedia.org
    ProxyCommand none
    ControlMaster auto

Host *.wikimedia.org
    User your_username_here
    IdentitiesOnly yes
    IdentityFile ~/.ssh/your_production_ssh_key
    ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org

## Internal Zones
Host *.wmnet
    User your_username_here
    IdentitiesOnly yes
    IdentityFile ~/.ssh/your_production_ssh_key

Host *.eqiad.wmnet
    ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org

Host *.codfw.wmnet
    ProxyCommand ssh -a -W %h:%p bast2001.wikimedia.org

Host *.esams.wmnet
    ProxyCommand ssh -a -W %h:%p bast3002.wikimedia.org

Host *.ulsfo.wmnet
    ProxyCommand ssh -a -W %h:%p bast4001.wikimedia.org

## Networking Equipment
Host *-eqiad.wikimedia.org *-eqord.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast1001.wikimedia.org

Host *-codfw.wikimedia.org *-eqdfw.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast2001.wikimedia.org

Host *-esams.wikimedia.org *-knams.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast3002.wikimedia.org

Host *-ulsfo.wikimedia.org
    ProxyCommand ssh -a -W %h:%p bast4001.wikimedia.org

Other tips

Debugging

If your production access has been approved but you aren't able to log in, you can ask for help in the Phabricator ticket for your access request. If you got access a long time ago and it's a new problem, you can file a new ticket and tag it with #operations.

Wherever you ask for help, make sure you include your SSH configuration (but not your key itself!) and the output you get when you run your ssh command with the -v option (verbose mode).

Notes

  1. 1.0 1.1 The form automatically adds the ticket to the Ops-Access-Requests project so the Operations team will see your request. It also creates a hidden subtask which is used to give the request a security review. The review may include technical details about sensitive systems, so it's kept hidden.
  2. You can also put your public key on your wiki user page, in a Phabricator paste, or in a Gerrit patchset you upload, but you can't include it in an email reply to the task.
  3. This protects against email spoofing.
  4. 4.0 4.1 If you request any level of sudo privileges, your request must have a security review at a weekly operations meetings. Sudo access is granted on an extremely limited basis, and will typically apply to the smallest permissions possible (user/process restricted over all). Expect this process to take at least one business week.
  5. Your manager's approval is usually not required, as you've already been granted access to the cluster; the project lead of the cluster you request access to should sign off (if in doubt, ask the Ops Clinic Duty person for the week.)