Fundraising/techops/procedures/servers-procurement and install

From Wikitech

Server Procurement and Install

This procedure is intended to dovetail with Server_Lifecycle.

Server Procurement

  1. FR Tech Ops or Datacenter Operations creates a procurement task in Phabricator using the SRE Procurement Request template, typically based on specific equipment that was approved in the annual CAPEX budget.
  2. The task is assigned to FR Tech Ops to verify the configuration and add instructions for hardware and network setup.
  3. FR Tech Ops documents the new hostname and reserved IP address in puppet/hieradata/hosts.yaml, in a commented-out host block.
  4. The task is reassigned to Datacenter Operations.
  5. Datacenter Operations obtains quotes from hardware vendors and assigns the task to FR Tech Ops to approve the purchase.
  6. FR Tech Ops approves the quote and assigns the task back to Datacenter Operations.
  7. Datacenter Operations orders and receives the server.
  8. Upon receipt at the datacenter, Datacenter Operations closes the procurement task and creates a new task is created to track installation and configuration.

Datacenter Install and Configuration

Tasks in this section are all handled by DC Ops.

  1. Configure DNS for the primary and management hostnames.
  2. Rack and cable the server according the specs on the procurement task. All servers have redundant power and network connections.
  3. Configure connected network switch ports, placing them in the correct VLAN per the specs on the procurement task.
  4. Set a temporary non-default management interface password.
  5. Apply any needed firmware updates
  6. Configure standard BIOS settings for console redirection, etc.
  7. Reassign the task to FR Tech Ops.

Software Install and Configuration

Tasks in this section are handled by FR Tech Ops unless otherwise noted.

  1. Add the new host to PFW policy in srx_config_generator/conf and use srx_config_generator/update_srx_webdrop to push the new policy to the upload directory on the PFW.
  2. Create a private Phabricator task for Network Operations to deploy the PFW policy changes. In the task describe the intent of the changes and include the Bug line linking the firewall policy deployment task to the server install task.
  3. Add new the host to iptables policy in puppet-private/secrets/iptables and deploy the new policy.
  4. Network Operations reviews and deploys the PFW policy.
  5. Log into the new server's management interface to set the management interface password and to obtain the network MAC address.
  6. In (frack) puppet/hieradata/hosts.yaml there should already be a commented-out host block for the new server. Add the MAC address to this block, and uncomment it.
  7. Deploy the hosts.yaml change and wait until the new configuration is applied to the DHCP server.
  8. SSH to the new server's management interface and pxeboot it.
  9. Use the management interface virtual console to select an install image from the pxeboot menu. New servers should be imaged with the latest supported OS release in use for the server's role.
  10. Once the install is complete, log into the system using the temporary new_install SSH key, and do a first-time puppet agent run to generate a puppet client cert.
  11. Log into the puppet CA server and use the puppet command to sign the new client certificate.
  12. Do another puppet agent run on the new server. Puppet will install the correct software and services, remove temporary and default credentials, and configure accounts appropriate for the server's role.
  13. When the puppet run is complete, reboot the server.
  14. At this point the server is hardened and ready for service.
  15. If an public IP is required, choose an available IP and create a Phabricator task for Network Operations to configure NAT for the new IP.
  16. Network Operations configures NAT.

Service Setup

Tasks in this section are handled by FR Tech Ops unless otherwise noted.

  1. Generate service SSH keys as necessary and add them to puppet-private.
  2. Copy over any necessary application data.
  3. Log into the primary puppetmaster and run rsync_blaster as necessary to deploy custom software and configuration.
  4. Verify that the appropriate services are operational.
  5. Add the new server to Icinga monitoring, in (operations) puppet/modules/icinga/templates/nsca_frack.cfg.erb
  6. SIte Reliability Engineering will review and deploy the change to the production monitoring system.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Fundraising/techops/procedures/servers-procurement_and_install&oldid=2144036"