Portal:Cloud VPS/Admin/Openstack source

From Wikitech
Jump to navigation Jump to search

This page describes how we fetch Openstack source code and install it to our machines.

This can be different depending on the deployment, component (horizon, neutron, nova, etc), release (mitaka, newton, etc) and target operating system (jessie, stretch, buster, etc).

From Debian packages

The majority of the Openstack we use comes from Debian packages.

In the old/realy days of Openstack, all releases were provided by a public apt repository by Mirantis Inc. This is no longer the case, but Debian/Ubuntu are still reference platforms for running Openstack, so complete suites of packages can be found ready to install.

In order to discover which openstack version are in official Debian repositories, one can check some useful links:

Version matrix

Locating the right Openstack version for the right target operating system may not be obvious at first sight.

Openstack release Openstack version number DEB repository Compatible Debian version Notes
Mitaka 8 jessie-backports Debian repo Debian Jessie, Debian Stretch Packages are in our reprepro, component 'openstack-mitaka-jessie'
Newton 9 stretch Debian repo Debian Stretch
Ocata 10 stretch-backports Debian repo Debian Stretch, Debian Buster Also in deb http://debian.ultimum.io debian-stretch ocata ocata-backports
Pike 11 <TBD> <TBD>
Queens 12
Rocky 13 buster Debian repo Debian Buster
Stein 14 experimental Debian repo
Openstack release Openstack version number DEB repository Compatible Debian version Notes

Server vs Client packages

We differentiate between clients and server packages.

  • clients are usually python libraries and utilities that interact with each component restful API.
  • server packages provide the daemons and tooling required to run the service itself (including the APIs).

Most of client software components are installed by means of Debian packages.

puppet code organization

Properly handling the version matrix in puppet code can be challenging. The layout we are trying to use right now is outlined here.

There are 2 main profiles that should be declared by each openstack deployment:

  • profile::openstack::base::clientpackages
  • profile::openstack::base::serverpackages

Usually, these profiles are just declared in the deployment-specific override, example:

  • profile::openstack::eqiad1::clientpackages --> declares --> profile::openstack::base::clientpackages
  • profile::openstack::eqiad1::serverpackages --> declares --> profile::openstack::base::serverpackages

This profile is responsible for calling the right openstack version/debian version module which contains the actual configuration, for example:

  • profile::openstack::base::clientpackages --> declares --> openstack::clientpackages::${version}::${::lsbdistcodename}
  • profile::openstack::base::serverpackages --> declares --> openstack::serverpackages::${version}::${::lsbdistcodename}

Here, ${version} is the openstack release in use, which is a different hiera key per deployment.

Each concrete module contains the specifics bits to be able to install the right packages in the system.

In case there are shared bits for client and server packages, these should be placed in the openstack::commonpackages::${version} class, and this clase should be declared require in the callers. Example:

  • openstack::serverpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka
  • openstack::clientpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka

Here is a summary list of things that can be included in each class:

  • profile::openstack::eqiad1::serverpackages call to the base profile, eqiad1 server package configuration override if required
  • profile::openstack::base::serverpackages call to the inner class, global server package configuration overrides if required
  • openstack::serverpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common server package installations
  • profile::openstack::eqiad1::clientpackages call to the base profile, eqiad1 client package configuration override if required
  • profile::openstack::base::clientpackages call to the inner class, global client package configuration overrides if required
  • openstack::clientpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common client package installations
  • openstack::commonpackages::${version} stuff that is common to both of the above (factorization)

Then, How are component packages really installed? compontents are installed by its own profile/class path, that installs and configures it. Example:

  • profile::openstack::eqiad1::nova::compute::service
    • profile::openstack::base::nova::compute::service
      • openstack::nova::compute::service
        • openstack::nova::compute::service::mitaka::stretch
          • (require) openstack::serverpackages::mitaka::stretch
            • (require) openstack::commonpackages::mitaka

And, Is such level of indirection / abstraction really needed?
We had really hard times in the past dealing with the version matrix. We were forced to adopt this layout to be able to manage the puppet code for each component/openstack release/debian version sanely.
The level of indirections and abstraction is the necessary evil.

importing packages into our reprepro

In some cases, .deb packages aren't always available. Example is the archival of some Debian backports repository. This already happened, for example in T216497 - CloudVPS: workaround archival of jessie-backports repo.

A common and safe workaround is to import the packages into reprepro.

This is valid in cases such as:

  • we don't trust the availability of the original repository
  • the original repository is about to be deleted for some reason
  • a given repository doesn't contain all the openstack suite

In all these cases, it's advisable to collect and import all openstack packages into reprepro.

From source code

There are some components that are deployed/used directly from upstream source code, without Debian packages being involved. This is usually horizon.

TODO: fill me.

See also