Wikimedia Cloud Services team/EnhancementProposals/ceph client refactor

This page contains a puppet refactor proposal for the ceph client code.

Problem statement

The ceph rbd client code in puppet does not have a easy and single way to use it, some points:

There's 3 ways of using it so far, sometimes it splits by host that uses it (ceph::client::rbd_cloudcontrol) sometimes by service (ceph::client::rbd_glance/rbd_libvirt) and for cinder it's in the main class instead (p:openstack::codfw1dev::cinder).

There is no clear way of setting up ceph rbd config/credentials for a given ceph pool/service, and we have at least 4:
- nova VM disks (often referred to as 'compute' in the puppet tree)
- glance images
- cinder volumes
- radosgw

Setting up config/credentials should be paired with the actual user/keydata being added in the ceph cluster. This is not something we can/want to do with puppet though, and automating this is something we can do in a later iteration.

Proposed solution

In summary: we should refresh the code to support the mappings:

Where:

openstack deployment is one of eqiad1 or codfw1dev for now
service is one of glance, cinder, nova, cinder-backups, ...
ceph deployment is currently "ceph", though this should be extended in the future to allow multi-dc/multi-cluster deployments (task)
access type this would be an abstraction over actual ceph auth permissions. Something like glance_client, nova_client, radosgw_client, osd_server, mon_server, ... this will then realize to the correct auth caps statements (ex. [osd] profile rbd pool=equiad1-compute).

abstraction

Inspired by how acme-chief manages certificates:

the source of truth is a private hiera hash with this layout, what it is called a DC-specific configuration hash:

profile::ceph::auth::load_all::configuration:
  cinder:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
  glance:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"
  osd:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        mds: "whatever"
        mgr: "whatever"
        mon: "whatever"
        osd: "whatever"

the hiera hash is per-DC (i.e, eqiad, codfw). Future iterations may evolve this to support multiple ceph farms per DC, but we don't want this at this point
clients may declare which creds they want by using an additional hiera key, for example, on file hieradata/roles/codfw/wmcs/openstack/codfw1dev/control.yaml:

profile::ceph::auth::deploy::selected_creds:
  - cinder
  - glance

Once this hiera structures are in place, we have 2 important profiles/modules to handle that data:

profile::ceph::auth::load_all: profile loaded on mons, reads the hiera hash (has a parameter 'configuration'), and calls internally the inner module:
- ceph::auth::load_all receives a list of credentials and loads them to the ceph running system.
profile::ceph::auth::deploy: profile loaded on ceph clients, like osd, cloudcontrol and friends. Reads configuration hash (has a parameter 'auths'), and calls internally the inner module:
- ceph::auth::deploy: deploys a keyring file on the filesystem.

code organization proposal

With the abstraction described above, we could use the puppet code like this:

* role::wmcs::ceph::mon
** profile::ceph::auth::load_all <-- reads DC-specific configuration hiera hash (i.e, eqiad or codfw)

* role::wmcs::ceph::osd
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files

* role::wmcs::openstack::codfw1dev::control
** profile::ceph::auth::deploy <-- reads DC-specific configuration hiera hash, and an array of individual keyfile names to deploy as files

A code example of this:

class profile::ceph::auth::load_all (
  Hash $configuration = lookup('ceph::auth'),
) {

   class { 'ceph::auth::load_all':
     configuration => $configuration,
   }
}

class ceph::auth::load_all (
  Hash $configuration,
) {
    # for each entry in the configuration hash
    # load it using 'ceph auth get-or-create'
}

Also:

class profile::ceph::auth::deploy (
  Hash $configuration = lookup('ceph::auth'),
  Array[String] $selected_creds = lookup('profile::ceph::auth::deploy::selected_creds'),
) {
  class { 'ceph::auth::deploy':
      configuration = $configuration,
      selected_creds = $selected_creds,
  }
}

class ceph::auth::deploy(
  Hash $configuration,
  Array[String] $selected_creds,  
) {
   # filter configuration for the selected creds
   # if we find a match, then create a keyring file resource with the keydata
}

Example hiera configuration:

File on private repository hieradata/common/codfw.yaml (which should be available to all systems on the DC --- TODO: we may want to have a more suitable place for this)

ceph::auth:
  cinder:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"
  glance:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"
  osd:
    keydata: XXXXXX
    keyring_filename: /path/to/file.keyring
    caps:
        caps_mds: "whatever"
        caps_mgr: "whatever"
        caps_mon: "whatever"
        caps_osd: "whatever"

File on the public repo hieradata/role/codfw/wmcs/openstack/codfw1dev/control.yaml, which should be available to all openstack codfw1dev cloudcontrol servers:

profile::ceph::auth::deploy:
  - cinder
  - glance

File on the public repo hieradata/role/codfw/wmcs/openstack/codfw1dev/virt_ceph.yaml, which should be available to all openstack codfw1dev virt servers:

profile::ceph::auth::deploy:
  - compute

Problem statement

Proposed solution

abstraction

code organization proposal

See also