Server Lifecycle

From Wikitech
Jump to: navigation, search

This page describes the lifecycle of Wikimedia servers, starting from the moment we acquire them and until the time we unrack them. A server has various states that it goes through, with several steps that need to happen in each state. The goal is to standardize our processes for 99% of the servers we deploy or decomission and ensure that some necessary steps are taken for consistency, manageability & security reasons.

Server states

Requested

  • New hardware is requested for use via the instructions on Operations_requests#Hardware_Requests.
  • Hardware Allocation Tech will review request, and detail on ticket if we already have a system that meets these requirements, or if one must be ordered.
  • If further details are needed, the task will be assigned back to requester for additional information.
  • If hardware is already available and request is approved by operations management, system will be allocated, skipping the rest of this process to the Existing System Allocation step.
  • If hardware must be ordered, the buyer will gather quotes from our approved vendors & perform initial reviews on quote(s).
  • At this time, quotes are still in RT
  • Technical review is done by operations team members familiar with hardware in question (see below), attaching their confirmation or corrections.
  • If there are corrections, ticket goes back to buyer and requester as needed until issues are clarified otherwise escalate to Systems Architect(s).
  • System Architect(s) perform final technical review to ensure the technical correctness, cost effectiveness, & architecture/roadmap alignment; attaching confirmation or corrections.
  • If there are corrections, ticket goes back to buyer and requester as needed until issues are clarified otherwise escalate to Operations Management.
  • Operations Management reviews ticket and attaches approval(s) or questions as needed & assigns ticket back to buyer.
  • Buyer may create an on-site hardware confirmation task, this ticket will confirm all parts, cables, and assorted items are available for incoming system.
  • Ticket is assigned to on-site tech, who must confirm or request the required hardware accessories needed to support the system.
  • Order may proceed even if all hardware is not on site, depending on missing hardware and lead times.

Existing System Allocation

  • Only existing systems (not new) use this step if they are requested.
  • If a system must be ordered, please skip this section and proceed to Ordered section.
  • System name is changed if required.
  • All puppet files are ensured to be absent of old server declarations.
  • System should have already been cleared out of monitoring.
  • If system listed in puppet decommission files to clear said monitoring, it should be removed from those files before it is allocated again.
  • All old system keys are confirmed to be removed from puppet master.
  • All ssh authorized
  • If all the above are good, the Hardware Allocation Tech will update your RT ticket and the Server Spares page to reflect the allocation.
  • Skip following steps until the Installation section.

Ordered

  • Only new systems (not existing/reclaimed systems)
  • Buyer purchases hardware, attaching ordering details to ticket.
  • Once order ships, buyer places inbound shipment ticket with datacenter vendor. Point of Contact details here.
  • Buyer assigns RT procurement ticket to the on-site technician to receive in hardware.

Post Order

  • An installation/deployment task should be created (if it doesn't already exist) for the overall deployment of the system/OS/service & place in the #operations project.
  • You can include the following steps on this ticket for ease of reference (taken from the entirely of the lifecycle document):
 System Deployment Steps:
  [] - mgmt dns entries created/updated (both asset tag & hostname) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - system bios and mgmt setup and tested [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - network switch setup (port description & vlan) [link sub-task for network configuration here, sub-task should include the network project]
  [] - production dns entries created/updated (just hostname, no asset tag entry) [link sub-task for on-site work here, sub-task should include the ops-datacenter project]
  [] - install_server module updated (dhcp and netboot/partitioning) [done via this task when on-site subtasks complete]
  [] - install OS (note jessie or trusty) [done via this task when network sub-task(s) complete]
  [] - accept/sign puppet/salt keys [done via this task post os-installation]
  [] - service implementation [done via this task post puppet/salt acceptance]

Receiving Systems On-Site

  • Before the new hardware arrives on site, a shipment ticket must be placed to the datacenter to allow it to be received.
  • If the shipment has a long enough lead time, the buyer should enter a ticket with the datacenter site. Note sometimes the shipment lead times won't allow this & a shipment notification will instead be sent when shipment arrives. In that event, the on-site technician should enter the receipt ticket with the datacenter vendor.
  • New hardware arrives on site & datacenter vendor notifies us of shipment receipt.
  • Packing slip for delivery should list an RT # & the RT ticket should have been assigned to the on-site technician for receipt at this time.
  • Open boxes, compare box contents to packing slip. Note on slip if correct or incorrect, scan packing slip and attach to ticket.
  • Compare packing slip to order receipt in the RT ticket, note results on ticket.
  • If any part of the order is incorrect, reply on RT ticket with what is wrong, and assign the ticket to the buyer on the ticket.
  • If the entire order was correct, please note on the procurement ticket. Unless the ticket states otherwise, it can be resolved by the receiving on-site technician at that time.
  • Assign asset tag to system, enter system into Racktables immediately, even if not in rack location.
  • Some systems will have a hostname assigned to them at time of order (usually for clustered systems), if it has been assigned, it will be on the procurement ticket. If it has not been assigned, name the systems in racktables with the Asset Tag under server name, and leave visible label blank.
  • Entry into racktables should always include the following: asset tag, OEM Serial Number (or service tag), hardware type (dropdown), procurement rt#, purchase date (on order), hardware warranty expiration, and location of system (tag).
  • The location tag lets us see whats in a location but NOT racked, so please make sure to check which site for each system.
  • Hardware warranties should be listed on the order ticket, most servers are three years after ship date.
  • Network equipment has one year coverage, which we renew each year as needed for various hardware.

Racked

  • A Phabricator task should exist with racking location and other details; made during the post-order steps above.
  • Hostname may be assigned (or system may refer to asset tag name until it is allocated for specific role)
  • Please see Server naming conventions for details on how hostnames are determined.
  • If hostname was not previously assigned, a label with name must be affixed to front and back of server.
  • DNS is updated for the mgmt network connections.
  • DNS for mgmt should include both the assettag.mgmt.site.wmnet as well as hostname.mgmt.site.wmnet.
  • DNS for production network will be set only for hostname, since a system will have a hostname before going on the production network, systems may not have this set if their usage isn't yet determined.
  • Racktables entry updated to reflect rack location.
  • System Bios & out of band mgmt setttings are configured at this time..
  • On-site Tech should fully test the mgmt interface to ensure it responds to ssh, they are able to login, reboot the system, and watch a successful BIOS POST over serial console.
  • Switch port(s) are assigned and labeled.
  • Label with hostname, if not available label with asset tag.
  • VLAN assignment is completed at this time only if system role is known.
  • After systems have been racked, if they are not immediately allocated to a service (IE: they are spare), a ticket should be created in core-ops and assgined to the HW Allocation Tech with the asset tags, so they can add to spares list.

Installation

  • Hostname must be assigned at this point.
  • Please see Server naming conventions for details on how hostnames are determined.
  • If hostname was not previously assigned, a label with name must be affixed to front and back of server.
  • DNS setup for production network.
  • $assettag.mgmt.$loc.wmnet should have been setup when the system was racked.
  • set up $name.mgmt.$loc.wmnet to the same IP as $assettag.mgmt.$loc.wmnet.
  • VLAN: Network port is set to proper vlan (and labeled with hostname if not yet labeled.)
  • Folks who can handle vlan assignments: Chris J, Faidon L, Mark B, Rob H.
  • Any ops folks who want this ability should speak to our network admins.
  • DHCP: Add server to appropriate file in Puppet, based on serial console port and speed:
  • modules/install_server/files/dhcpd/linux-host-entries.ttyS0-9600 = com port 1, speed of 9600
  • modules/install_server/files/dhcpd/linux-host-entries.ttyS0-115200 = com port 1, speed of 115200
  • modules/install_server/files/dhcpd/linux-host-entries.ttyS1-115200 = com port 2, speed of 115200 (most hosts)
  • You can pull this information from the management of most systems, as described in their specific pages under Platform-specific documentation.
  • Decide on partition mapping & add server to modules/install_server/files/autoinstall/netboot.cfg
  • Detailed implementation details for our Partman install exist here.
  • The majority of systems should use automatic partitioning, which is set by inclusion on the proper line in netboot.cfg.
  • Any hardware raid would need to be setup manually via rebooting and entering raid bios.
  • Right now there is a mix of hardware and software raid availability.
  • File located @ puppet modules/install_server.
  • partman recipe used located in modules/install_server
  • Please note if you are uncertain on what to pick, you should lean towards LVM.
  • Many reasons for this, including ease of expansion in event of filling the disk.
  • Proceed with Installation
  • Reboot system and boot from network / PXE boot.
  • acquires hostname in DNS
  • acquires DHCP/autoinstall entries
  • gets installed

Post-Install: Get puppet running

  • Nothing replaces fully understanding how our deployment of Puppet operates, as detailed on the service info page.
  • Warning: if you are rebuilding a pre-existing server (rather than a brand new name), on the puppet master (puppetmaster1001), run puppet cert destroy $server_fqdn to clear out the old certificate before beginning this process. If you already began, also run (on the server you're building, not the puppet master) find /var/lib/puppet/ssl -type f -exec rm {} \; to clean out the client.
  • Login to the puppet master (puppetmaster1001).
  • from puppetmaster1001, sudo /usr/local/sbin/install-console $server_fqdn to log into $server
  • on $server, run puppet agent --test
It should whine that it can't get its cert automatically: Exiting; no certificate found and waitforcert is disabled
  • on the puppet master (puppetmaster1001), run puppet cert -l to list all pending certificate signings.
  • on the puppet master, run puppet cert -s $server_fqdn for the specific server you wish to sign keys for.
  • Now again on $server, run puppet agent --enable to administratively enable puppet, and then puppet agent --test. It should now succeed.
  • Now that puppet has run, sign the new salt key on the salt master (neodymium): salt-key -a $server_fqdn
  • After your first couple of successful puppet runs, you should reboot just to make sure it comes up clean.
  • Your host should now appear in puppet stored configs and therefore in ganglia and icinga.
  • Important: check that salt-minion is properly running on the host. If your host does not include the base or standard classes from puppet, include the specific class for the salt minion (at this writing role::salt::minion).
  • Look to see if the process 'salt-minion' is running.
  • Check /var/log/salt/minion for messages about keys; if you see continuing messages "The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate", you need to accept the key on the salt master host, see Salt for how to do this.

In Service

  • Where all servers aspire to be.

Reinstallation

  • The wmf_auto_reimage.py is a script that can help you automate some of the tasks below. YOU SHOULD USE THIS SCRIPT FOR REINSTALLS, as it prevents you from missing steps. It needs to be run from the Salt master (neodymium as of now) with sudo from a screen session. See also the help with sudo wmf-auto-reimage -h. The old wmf-reimage script is still used as part of the wmf-auto-reimag one. List of features of the wmf_auto_reimage script:
    • depool the host via conftool before reimaging
    • run the apache-fast-test on the host after the reimage
    • update a Phabricator task with the reimage info
  • A Phabricator ticket should be created detailing the reinstallation in progress.
  • System services must be confirmed to be offline. Checking everything needed for this step and documenting it on this specific page is not feasible at this time. Please ensure you understand the full service details and what software configuration files must be modified. This document will only list the generic steps required for the majority of servers.
  • The following instructions assume the system is online and responsive. If the system is offline, simply skip the steps requiring you do run something on the host.
  • If server is part of a service pool, ensure it is set to false or removed completely from pybal/LVS.
  • Instructions on how to do so are listed on the LVS page.
  • If server is part of a service group, there will be associated files for removal or update. The service in question needs to be understood by tech performing the decommission (to the point they know when they can take things offline.) If assistance is needed, please seek out another operations team member to assist.
  • Example: db class machines are in associated db-X.php, memcached in mc.php.
  • Remove server entry from DSH node groups, if present (if the server is part of a service pool, this is most probably not necessary as dsh gropups are populated from conftool)
  • These files are maintained in operations/puppet:hieradata/common/scap/dsh.yml
  • Put the downtime into icinga if said downtime will generate pages.
  • Disable puppet on the host (puppet agent --disable)
  • Admin log whenever you disable or enable puppet on a host!
  • Manually revoke keys from puppet.
  • Instructions on how to do so are on the Puppet service details page.
  • $ puppet cert clean <fqdn>
  • Power down system.
  • Manually revoke keys from salt.
  • Instructions on how to do so are on the Salt service details page.
  • $ salt-key -d <fqdn>
  • After the above is done, system installation can proceed from the Installation section above.

Reclaim to Spares OR Decommission

Steps for ANY Opsen

  • A Hardware requests ticket should be created detailing if system is being decommissioned (and removed from datacenter) or reclaimed (wiped of all services/data and set system as spare for reallocation).
  • System services must be confirmed to be offline. Checking everything needed for this step and documenting it on this specific page is not feasible at this time(but we are working to add them all). Please ensure you understand the full service details and what software configuration files must be modified. This document will only list the generic steps required for the majority of servers.
  • Disable ALL service level checks in icinga for host.
  • If server is part of a service pool, ensure it is set to false or removed completely from pybal/LVS.
    • Instructions on how to do so are listed on the LVS page.
  • If server is part of a service group, there will be associated files for removal or update. The service in question needs to be understood by tech performing the decommission (to the point they know when they can take things offline.) If assistance is needed, please seek out another operations team member to assist.
    • Example: db class machines are in associated db-X.php, memcached in mc.php.
  • Remove server entry from DSH node groups (if any).
    • If the server is part of a service group, common DSH entries are populated from conftool, unless they're proxies or canaries
    • The list of dsh groups is in operations/puppet:hieradata/common/scap/dsh.yaml.
  • Remove system entries in site.pp, replace with system entry for role::spare, merge changes.
  • Remove all hiera data entries for host.
  • Run puppet on host to be reclaimed/decommissioned.
    • Leaving the host on, but with role::spare will allow it to receive security updates.

Steps for DC-OPS

  • Confirm all puppet manifest entires removal, DSH removal, Hiera data removal.

These steps, once started, must be completed without interruption.

  • Disable puppet on the host (puppet agent --disable)
    • Admin log whenever you disable or enable puppet on a host!
  • Remove all references in puppet:
    • remove from site.pp and from hiera data (both individual host files and entries in regex.yaml, if any
    • remove from netboot.cfg (puppet:///modules/install_server/files/autoinstall/netboot.cfg)
    • remove from DHCPD lease file (puppet:///modules/install_server/files/dhcpd/linux-host-entries.ttyS... filename changes based on serial console settings)
    • Instructions on how to do so are on the Puppet service details page.
    • $ puppet node clean <fqdn>
    • $ puppet node deactivate <fqdn>
      • These 2 commands immediately preceding this should also remove the host from Icinga monitoring:
    • Run puppet on the icinga master (currently einsteinium.wikimedia.org), so that all alerts for the host are removed
    • Alternatively, put the host and all services into downtime for 1+ day, as it will then not alert when the host is powered down & the next puppet run on the icinga host will remove it from monitoring.
  • Manually revoke keys from salt.
    • Instructions on how to do so are on the Salt service details page.
    • $ salt-key -d <fqdn>
  • disable host's port on switch.
    • If system is being reclaimed for spare, do not change port label.
    • If system is being decommissioned, please do not wipe port description until AFTER it is unracked.
    • THIS MUST BE DONE, or host can be powered up and will be network accessible (but not in puppet and not getting security updates)
    • If you can't get puppet to run happily on icinga, get help. If help is not available, stop, renable puppet on the host, and start these steps again once you can get help with icinga.
    • You should either ensure monitoring is removed, or at minimum disable notifications for that host. Don't generate paging alerts for systems you are decommissioning.
  • Power down system.

End steps that must be completed without interruption. The following can be done one at a time and/or with long breaks in bewtween.

  • Remove DNS entries for the production network.
    • Don't remove the mgmt DNS entries at this time!
      • Reclaims never have mgmt entries removed, and decom servers should keep them until they are wiped and unracked.
  • Update associated Phabricator ticket, detailing steps taken and resolution.
    • If system is decommissioned by on-site tech, they can resolve the ticket.
    • If system is reclaimed into spares, ticket should be assigned to the HW Allocation Tech so he can update spares lists for allocation.

Decommission Specific

  • A Phabricator ticket for the decommission of the system should be placed in the #hardware-request project and the appropriate datacenter-specific ops-* project.
  • All further decommission steps are handled by the on-site technician.
  • Wipe all disks on system with minimum of 3 passes.
  • We presently boot off USB version of DBaN.
  • Reset all system bios, mgmt bios, & raid bios settings to factory defaults.
  • Unrack system & update Racktables, moving system from rack location to decommissioned rack.
  • Unless another system will be placed in the space vacated immediately, please remove all power & network cables from rack.
  • Once server is un-racked, do the following:
  • Remove its mgmt DNS entries.
  • Remove port description label for decomissioned host's switch port

Position Assignments

The cycle above references specific position/assignments, without referring to name. To keep the document generic, we'll keep the cycle with positions listed, and just list those folks here.

  • Buyer / HW Allocation Tech: Rob H (US), Mark B (EU)
  • On-site Tech EQIAD: Chris J
  • On-site Tech CODFW: Papaul T
  • On-site Tech ULSFO: Rob H
  • Director Technical Operations : Mark B
  • Operations Technical Review: Mark B, Faidon L