Jump to content

Portal:Cloud VPS/Admin/Openstack source

From Wikitech
This page is currently a draft.
More information and discussion about changes to this draft on the talk page.

This page describes how we fetch Openstack source code and install it to our machines.

This can be different depending on the deployment, component (horizon, neutron, nova, etc), release (mitaka, newton, etc) and target operating system (jessie, stretch, buster, etc).

From Debian packages

The majority of the Openstack we use comes from Debian packages.

In the old/realy days of Openstack, all releases were provided by a public apt repository by Mirantis Inc. This is no longer the case, but Debian/Ubuntu are still reference platforms for running Openstack, so complete suites of packages can be found ready to install.

In order to discover which openstack version are in official Debian repositories, one can check some useful links:

Version matrix

Locating the right Openstack version for the right target operating system may not be obvious at first sight.

Openstack release Openstack version DEB repository Compatible Debian version Notes
Mitaka 13 jessie-backports Debian repo, or:

deb http://osbpo.debian.net/debian jessie-mitaka-backports main
deb http://osbpo.debian.net/debian jessie-mitaka-backports-nochange main

Debian Jessie, Debian Stretch Packages are in our reprepro, component 'openstack-mitaka-jessie'
Newton 14 Stretch Debian mirrors Stretch
Ocata 15 deb http://osbpo.debian.net/debian ocata main Stretch Packages are in our reprepro, component 'openstack-ocata-stretch'
Pike keystone 12

nova 16

neutron 11

glance 15

designate 5

deb http://osbpo.debian.net/debian/ stretch-pike-backports main

deb http://osbpo.debian.net/debian/ stretch-pike-backports-nochange main

Stretch
Queens 17 deb http://osbpo.debian.net/debian stretch-queens-backports main

deb http://osbpo.debian.net/debian stretch-queens-backports-nochange main

Stretch
Rocky 18 Official Debian mirrors for Buster. For Stretch backports:

deb http://osbpo.debian.net/debian stretch-rocky-backports main
deb http://osbpo.debian.net/debian stretch-rocky-backports-nochange main

Stretch, Buster
Stein 19 deb http://osbpo.debian.net/debian buster-stein-backports main

deb http://osbpo.debian.net/debian buster-stein-backports-nochange main

Buster
Train 20 deb http://osbpo.debian.net/debian buster-train-backports main

deb http://osbpo.debian.net/debian buster-train-backports-nochange main

Buster
Openstack release Openstack version DEB repository Compatible Debian version Notes

Server vs Client packages

We differentiate between clients and server packages.

  • clients are usually python libraries and utilities that interact with each component restful API.
  • server packages provide the daemons and tooling required to run the service itself (including the APIs).

Most of client software components are installed by means of Debian packages.

puppet code organization

Properly handling the version matrix in puppet code can be challenging. The layout we are trying to use right now is outlined here.

There are 2 main profiles that should be declared by each openstack deployment:

  • profile::openstack::base::clientpackages
  • profile::openstack::base::serverpackages

Usually, these profiles are just declared in the deployment-specific override, example:

  • profile::openstack::eqiad1::clientpackages --> declares --> profile::openstack::base::clientpackages
  • profile::openstack::eqiad1::serverpackages --> declares --> profile::openstack::base::serverpackages

This profile is responsible for calling the right openstack version/debian version module which contains the actual configuration, for example:

  • profile::openstack::base::clientpackages --> declares --> openstack::clientpackages::${version}::${::lsbdistcodename}
  • profile::openstack::base::serverpackages --> declares --> openstack::serverpackages::${version}::${::lsbdistcodename}

Here, ${version} is the openstack release in use, which is a different hiera key per deployment.

Each concrete module contains the specifics bits to be able to install the right packages in the system.

In case there are shared bits for client and server packages, these should be placed in the openstack::commonpackages::${version} class, and this clase should be declared require in the callers. Example:

  • openstack::serverpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka
  • openstack::clientpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka

Here is a summary list of things that can be included in each class:

  • profile::openstack::eqiad1::serverpackages call to the base profile, eqiad1 server package configuration override if required
  • profile::openstack::base::serverpackages call to the inner class, global server package configuration overrides if required
  • openstack::serverpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common server package installations
  • profile::openstack::eqiad1::clientpackages call to the base profile, eqiad1 client package configuration override if required
  • profile::openstack::base::clientpackages call to the inner class, global client package configuration overrides if required
  • openstack::clientpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common client package installations
  • openstack::commonpackages::${version} stuff that is common to both of the above (factorization)

Then, How are component packages really installed? compontents are installed by its own profile/class path, that installs and configures it. Example:

  • profile::openstack::eqiad1::nova::compute::service
    • profile::openstack::base::nova::compute::service
      • openstack::nova::compute::service
        • openstack::nova::compute::service::mitaka::stretch
          • (require) openstack::serverpackages::mitaka::stretch
            • (require) openstack::commonpackages::mitaka

And, Is such level of indirection / abstraction really needed?
We had really hard times in the past dealing with the version matrix. We were forced to adopt this layout to be able to manage the puppet code for each component/openstack release/debian version sanely.
The level of indirections and abstraction is the necessary evil.

importing packages into our reprepro

In some cases, .deb packages aren't always available. Example is the archival of some Debian backports repository. This already happened, for example in T216497 - CloudVPS: workaround archival of jessie-backports repo.

A common and safe workaround is to import the packages into reprepro.

This is valid in cases such as:

  • we don't trust the availability of the original repository
  • the original repository is about to be deleted for some reason
  • a given repository doesn't contain all the openstack suite

In all these cases, it's advisable to collect and import all openstack packages into reprepro.

From source code

There are some components that are deployed/used directly from upstream source code, without Debian packages being involved. This is usually horizon.

TODO: fill me.

See also