Search/adding-nodes-to-deployment-prep

From Wikitech

Adding Nodes to Deployment-Prep

Deployment-Prep, aka "Beta Cluster" is our most volatile, least production-like environment (for the Search team, at least). Thus, this environment is where we begin any upgrade or config change process. Unfortunately, deployment is a largely manual process at the moment. See below for details.

Hack acknowledgement

This only works when the server's FQDN and the FQDN of the site to be hosted are exactly the same.

Commit to Puppet Repo

Add the new servers' fully-qualified domain names into the operations/puppet repo . Example commit

Horizon

Go one-at-a-time and repeat this process for each instance.

Once the code is committed to the puppet repo, launch the new instance from the horizon dashboard

  • Add an existing, valid cert to the hiera data for the newly-built instance. For example:


profile::elasticsearch::dc_settings:

certificate_name: deployment-elastic07.deployment-prep.eqiad.wmflabs

tls_port: 9243

puppetmaster: deployment-puppetmaster04.deployment-prep.eqiad.wmflabs

Newly-deployed host (puppet client)

Can't access the new host? This could happen if puppet fails. See When You Can't Login for recommended workarounds. Once you can login:

  • remove /var/lib/puppet/ssl
  • set deployment-puppetmaster04.deployment-prep.eqiad.wmflabs as puppet server in /etc/puppet/puppet.conf
  • run-puppet-agent

Puppet agent will send a certificate signing request (CSR) to the puppet master, deployment-puppetmaster04.deployment-prep.eqiad.wmflabs . Let's go there now!

Puppet Master (puppet server)

  • sign newly-generated cert (verify fingerprint): `puppet cert sign -i $fqdn`
  • copy newly-signed cert from` /var/lib/puppet/server/ssl/ca/signed/$fqdn.pem` to /var/lib/git/operations/puppet/modules/profile/files/ssl/$hostname.deployment-prep.eqiad.wmflabs.crt . Note the use of the old domain, deployment-prep.eqiad.wmflabs.crt !

Newly-deployed host (puppet client)

  • Copy the private key, located at /var/lib/puppet/ssl/private_keys/$fqdn.pem on the client, to the puppet master's private repo, /var/lib/git/labs/private/modules/secret/secrets/ssl/$hostname.deployment-prep.eqiad.wmflabs.key Again, note use of the old domain! Commit the key to the repo, but DO NOT push it out.

Horizon

Update the certificate_name value to the hostname of your newly-deployed host.

from: 

profile::elasticsearch::dc_settings:
  certificate_name: deployment-elastic07.deployment-prep.eqiad.wmflabs

to: 

profile::elasticsearch::dc_settings:
  certificate_name: $your_hostname_here.deployment-prep.eqiad.wmflabs

Again, be sure to keep the older deployment-prep.eqiad.wmflabs.crt domain.

Newly-deployed host (puppet client)

  • Manually install java package ( openjdk-11-jdk at the time of this writing)
  • run-puppet-agent again
  • reboot
  • run-puppet-agent yet again

At this point, your new host should have run puppet successfully, and all its services should be up and running. Verify by checking the health of the cluster and the nginx TLS frontend.

When you can't login

If puppet can't complete its run (common when testing changes), you probably won't be able to login. Suggested workarounds:

  • Via cloud-init, append your public key to /etc/ssh/userkeys/root. You'll need to check "configuration drive" when you deploy in horizon. You also may have to make the file append-only to keep puppet from overwriting it (chattr +a) . When cloud-init finishes (~10m after instance launch), you should be able to login via SSH as root.
  • (SRE only) You can also login via the virsh console. From the instance's page in horizon:
    • Find Host and Instance Name. Instance name will look like i-0006e018
    • Login to the host via SSH, then connect to virsh console: virsh console i-0006e018 . You will be logged to the console as root.
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Search/adding-nodes-to-deployment-prep&oldid=2004628"