Fundraising/techops/procedures/servers-server build standard

From Wikitech

Fundraising Server Build Standard

Maintainer: Fundraising Tech Operations

Created: 2023-07-21

Review Required by: 2024-07-25

Purpose

This build standard applies to servers in the Fundraising environment, which are installed, secured, and maintained by FR Tech Ops.

Server Lifecycle

Fundraising servers generally follow WMF's Server Lifecycle though the separate and isolated Fundraising server environment has different networking, puppet, and DNS procedures. During decommissioning, all disks and media are destroyed or erased before leaving the secure datacenter environment.

Operating System and Base Software

Fundraising servers are built with the latest stable Debian Linux long term support (LTS) release supported in our environment, meeting the WMF Infrastructure Foundations Operating System Upgrade Policy.

OS upgrades are handled in in phases, typically one server role at a time, as we are able to test and integrate configuration changes to support the new release.

OS releases are phased out of service before LTS support ends.

Server Role and Function

Generally a server has a single primary function (e.g. a database, mail, or application server) although in a few cases more than one function will coexist. These exceptions are cases where the functions are related and share the same security level. For example our CiviCRM application server hosts both the web application and scheduled queue consumer jobs.

Server Operating System

Wherever possible we use stock packages distributed and maintained as part of the OS release. In some cases we use vendor packages or locally build packages, if they are better suited for or no stock package exists.

Server Configuration and Hardening

Server hardening practices include the following:

  • extraneous accounts are removed or disabled
  • no vendor default passwords are in use
  • unnecessary software is removed
  • unnecessary services, protocols, daemons, and functions are removed or disabled
  • security parameters are configured to prevent misuse
  • openssh is enabled and restricted to public key authentication
  • a host firewall is enabled and configured appropriately for server role
  • intrusion detection is enabled and configured appropriately for server role
  • a virus scanner is configured and enabled for servers in the PCI Cardholder Data Environment
  • audit logging is enabled
  • logging is enabled to offhost central log collector
  • the host is configured to remain synchronized to network time
  • additional hardening is performed according to Debian benchmarks from Center for Internet Security

Software Updates

All servers are configured to use Debian and vendor repositories. Most servers have unattended-upgrades software running and install updates automatically. In some cases specific packages or servers are excluded from automation if it would potentially cause outages.

All servers run a script daily that uses a simulated dist-upgrade to determine if there are updates available for installed packages. This script notifies FR Tech Ops if manual intervention is required to bring a server up to date.

OS and software updates are applied as soon as possible after they become available, and within timeframes specified by WMF (draft) Security Policy - Vulnerability Matrix.

For more information on OS and software vulnerability management see Operating System and Vendor Software Vulnerability Procedures.

Testing

OS and configuration testing is performed in virtual machine clusters or containers running on individual SRE and developer computers. This testing helps us discover and resolve software and configuration changes that need to be addressed to support different OS releases in our environment.

Once an OS/software release is ready for production staging, an out-of-service server is reimaged and used in a canary deployment strategy.

Server Imaging and Orchestration

FR Tech Ops maintains a pxeboot install environment which uses Debian Preseed for initial server imaging. This environment supports installs from a selection of operating systems including the latest Debian LTS stable release and, if needed, a legacy release. The base image includes a preconfigured puppet-agent.

Puppet automates standardized server hardening and configuration. The puppet server configuration has been designed per-role according to benchmarks for Debian Linux from the Center for Internet Security, and is revision-controlled in a private git repository.

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Fundraising/techops/procedures/servers-server_build_standard&oldid=2094572"