SRE Team requests
Here's what you can do if you need help from the Wikimedia Site Reliability Engineering team.
Urgent issues are generally imminent risks to site security, like compromised SSH keys. Whichever route you choose, make sure to keep at it until you get confirmation that a member of the SRE team has received the message.
#wikimedia-operationsIRC channel (in an emergency, consider using the hotword
#pageto get attention)
- Phone (Foundation staff members have access to the contact list on Office wiki)
The majority of operations requests should be filed within the Wikimedia Phabricator installation using the #operations project tag.
- If you keep the default priority to 'Needs Triage' and it is in the
#Operationsproject, our SRE Clinic Duty assignee for the week will triage your request.
- This link will create a task in the operations project.
If you further refine your request using the below instructions, it will usually result in faster triage.
- Requesting shell access - fully documented instructions on how to gain shell access; required for any user requesting access.
- Google Search Console access - Access to the google search console.
Blocked by Ops
- If your particular project or task is currently blocked by operations, please detail such on the task (or sub-task) and flag it with the project tag Blocked-on-Operations.
- Our Ops Clinic Duty Dashboard lists these.
- This project #domains is for all domain registration requests, nameserver updates, and anything involving a domain registrar.
- Example: Volunteer transferring domain to WMF control.
- Example: Incoming domains needing implementation/support on cluster.
- This link will create a task in the domain & operations projects.
- It is advised that you leave your priority as 'Needs Triage' and not assign it to a specific person. This will result in it showing in the top of the operations triage lists.
- If you are requesting that Wikimedia register a domain that is currently unregistered, you will want to select option Security: Other confidential issue.
- This allows you, plus the wmf-nda to view the task, but not the entire internet; requesting we register an unregistered domain in an open task is a nice way to let squatters know what to register.
- TL;DR Click Hardware requests to file a task for requesting hardware. But please read the following.
- Requesting a server for your service should only occur after the following:
- You can click the link above to pre-populate a hardware request ticket with the basic fields for entry. Please also include the operations and hardware-requests projects. These include:
- Labs Project Tested, Site/Location, Number of systems, Service, Networking Requirements like access to specific networks, Processor Requirements, Memory, Disks:, NIC(s), Partitioning Scheme, and any other relevant notes/info.
- Note that Operations might suggest using a VM instead if it deems it applicable.
- Note that you don't place server requests on our procurement project.
- A single #hardware-request can generate multiple #procurement sub-tasks, as each sub-task could be pricing from a specific vendor.
Virtual machine requests (Production)
TL;DR. Click VM Requests and fill in the form. But please do read the following.
- THIS IS NOT TOOLFORGE.
- This is for requesting a virtual machine in the production cluster. (This is usually as an alternative to a bare metal server.)
- VMs are great for hardware sharing, increasing hardware usage. If your service does not have specific hardware requirements, a VM is an ideal candidate for it. But if it has critical performance requirements, it might very well not be.
- Requesting a server for your service should only occur after the following:
- You can click the link above to pre-populate a vm-request ticket with the basic fields for entry. These include:
- Cloud VPS Project Tested, Site/Location, Service, Networking Requirements, Processor Requirements, Memory, Disks, and any other relevant notes/info.
- Do note that Operations might suggest using bare metal hardware instead if it deems it necessary
- Networking wise, multiple NICs for throughput increase is not viable in a VM
- Disk performance is limited by the underlying technology and resource sharing.
Other Purchases: SSL Certificates, Support Contracts
- All other requests for operations purchasing of support contracts, ssl certificates, and other related items should be placed in the Procurement project
- Please note that mail aliases are not handled by SRE anymore. Mail aliases under the wikimedia.org domain are handled by the OIT team. Please send a mail to firstname.lastname@example.org to request one. Please note that if you are not staff, and require a mail alias, you should request it via your working group/team leads/technical mentor/staff.
- Only if you need an alias in another domain besides wikimedia.org or have a specific reason that you need it to trigger before Google routing, create a SRE request in the Operations project.
- If you have an existing exim mail aliases handled by SRE you are encouraged to move it by requesting the same from OIT and telling SRE to delete the existing one on their side. This would be part of T122144. Thanks!
- Please also see https://meta.wikimedia.org/wiki/Mailing_lists#Create_a_new_list
- The operations team doesn't create all mailing lists. Instead, you should file a general request under the Wikimedia-Mailing-lists project in Phabricator; please leave the priority as 'Needs Triage' for our our SRE Clinic Duty assignee to better notice it.
- Please include the following:
- requested name of the mailing list, ending in @lists.wikimedia.org
- reasoning/explanation of purpose (and link to community consensus, if applicable)
- initial list administrator's email address
- secondary list administrator's email address (as a backup)
- description of the list for the list info page (should include even if private list so ops and mailman admins know why it exists.)
- Note if this should be public or private, and if archives should exist or not. (If list is private, archives should be private.)
- General list administration is handled by an individual lists administrators; administrators can be viewed on the lists information page.
- Operations involvement is typically only required when a list administrator is not listed on the list information page, or if the administrator has become unavailable for the role.
- We will NOT simply change list owners; all attempts to handle the request via the usual means/admins must be exhausted. We will attempt to also contact the list administrator before we change anything.
- If you still want operations assistance, please file a task with both the #operations & #Wikimedia-Mailing-lists projects.
- This link will create a task in both operations & Wikimedia-Mailing-lists projects.
- Any patches that require an operations team member review should have a Phabricator task and have both the operations and Patch-For-Review project tags assigned to it.
- PuppetSWAT takes place twice weekly. Any simple patches can be included during its SWAT window. Please see page for further details.
- Network requests (router configuration, switch port descriptions, vlan assignments, etc) should have a Phabricator task and have both the operations and network project tags assigned to it.
- Please do not assign a specific team member for review unless they are the subject matter expert (though CCing them if you are uncertain is valid); otherwise our SRE Clinic Duty assignee will attempt to triage to the appropriate parties.
- This link will create a task with the operations and network projects associated with it.
- Subnets/VLANS are listed on the switches (not public accessible) and in our operations/dns git repo (public accessible).
- Schema changes on production databases have to be approved and applied by DBAs. Instructions on how to request its application are on the Schema change page.
- Please do not assign a specific team member for review (though CCing them if you are uncertain is valid); otherwise our Ops Clinic Duty assignee will attempt to triage to the appropriate parties.
- This link will create a task in the #Blocked-on-schema-change and #DBA projects
- Only use #Blocked-on-schema-change when the change is final, not while it is in progress/hasn't been reviewed
- Normal schema changes can take up to 2 weeks to take effect. Those involving key tables such as revision, page or image may take more.
- Operations team members idle in
- This is generally useful for vague questions or project planning, but non-ideal for hardware requests, access requests, or ongoing work.
- If the request will result in work on the part of the operations team member, a Phabricator task will be requested to track the work.
- There is an SRE Clinic Duty assignee from the operations team for every week.
- Clinic duty person is listed in the topic for
#wikimedia-operations, as well as on Ops Clinic Duty this changes every Monday.
- The clinic duty person can be pinged, and is the first point of contact in IRC for operations issues.
- Please note that our operations team works in multiple time zones, and the clinic assignee for any given week will likely be working within their own local time zone.
Ops mailing list
Ops team members are subscribed to email@example.com.