Portal:Cloud VPS/Admin/notes/Puppet Possibilities

Talking Puppet, Hiera, Openstack and process

Meeting Agenda

Old universe

• Nova in many places for (nearly) all things
• intermingled repo management
• Current install specific groupings (nova-api and nova-network, etc)
• Per host values for labtest in hiera and regex madness
• hiera lookups buried in: roles, modules, templates
• Overlapping requirements for components handled via require_package() a lot
• Passing dicts of huge numbers of unrelated params to get to a few
• Duplicate params at various levels of the hiera tree
• Convoluted component interdependence
• Associated configuration lumped into large blocks in roles or other

• ferm rules that make it difficult to reason on cleanup as it's unclear why it existed

• Inconsistent strategy on hostname => ip esp for ferm
• data structured mingled in with code (labsdb, etc)
• The same value would be named multiple things throughout a config tree

General

• Three failover scenarios

• labcontrol (host and cold) via hiera
• controlled via hiera params setting a boolean in a profile and passed to modules

  class openstack2::keystone::service(
      $active,
  

• labservices (partial host and cold)
• labnet (host and cold) role swapping <== looking to deprecate

• Deployment separations (deployments to come)

• deployments may end up being regions?
• deployments are made up of components
• components can be upstream openstack specific or not

• Why not use inheritance:

• https://puppet.com/docs/puppet/4.10/lang_classes.html#inheritance
• so much implicit reasoning already

• We are working on a holistics openstack / VPS product profile model
• consistent naming across components glance/service.pp, keystone/service.pp

• If it is parameterized per deployment is worth doing base and deployment profiles potentially under profiles/manifests/openstack

• Understanding our Hiera use cases:

• https://phabricator.wikimedia.org/T171494#3479169
• Hiera: per site, and our special base instead of defunct common. Hiera consistency must be our bread and butter.
• hiera_lookup

• Labs hiera tree is (also) a nightmare.

• expand_paths vs not
• labs/private is an overloaded dual purpose shim
• back end split
• Labs hiera is a nightmare
• Per deployment labs/private or equiv may be needed

• Puppet norms for new style are not always set up for sucess

• base_firewall, standard, remaining implicit override, hiera adding functionality, etc

Upstream to us Style

• https://wikitech.wikimedia.org/wiki/Puppet_coding#Coding_Style
• https://wikitech.wikimedia.org/wiki/Puppet_coding/Style_guide
• https://puppet.com/docs/puppet/5.3/style_guide.html
• I want to get more consistent with our own style

To Be Done

• Issue of cross site deployment variables (db1047)
• Issue of special failover with labnet
• firewall rules buried in site.pp
• tools stub zone on labservices persistence
• Issue of cleanup for hiera
• openstack2 => openstack
• test failover for pairs

Moving forward

• resolve() within ferm

• per ferm update resolution instead of per puppet run
• human readable in diffs

• Our own sub-style consistency that is an extension but not an override

• File and resource consistence etc
  • owner,group,mode,source,requires
• manifest flow: include, package, file(s), service

• This results in per deployment per component profiles and per deployment roles

• Much easier to test in labtest without effecting main, and we SHOULD/need to/must
• staging models mean more of an opportunity for productive meaningful refactor
• testplan designations

• "foo: bar for x". "testplan: baz?"
• modules/roles/manifests/wmcs

• potentially: modules/roles/manifests/wmcs/tenants

• Anything
• Functional approach to hiera and parameters. Ironic? :)
• In the mid-term we must consolidate on WMCS managed hiera strategy
• A large and intimidating Toolforge cleanup must happen
• We will have to solve the puppet multi-tenancy problem in the next 2 years or die trying