This page is currently a draft.
Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page.

This page describes how we fetch Openstack source code and install it to our machines.

This can be different depending on the deployment, component (horizon, neutron, nova, etc), release (mitaka, newton, etc) and target operating system (jessie, stretch, buster, etc).

From Debian packages

The majority of the Openstack we use comes from Debian packages.

In the old/realy days of Openstack, all releases were provided by a public apt repository by Mirantis Inc. This is no longer the case, but Debian/Ubuntu are still reference platforms for running Openstack, so complete suites of packages can be found ready to install.

In order to discover which openstack version are in official Debian repositories, one can check some useful links:

Debian snapshot archive: neutron keystone nova
Debian package tracker: neutron keystone nova
Debian maintainer dashboard: all packages version matrix
Debian QA overview: all packages version matrix

Version matrix

Locating the right Openstack version for the right target operating system may not be obvious at first sight.

Openstack release	Openstack version	DEB repository	Compatible Debian version	Notes
Mitaka	13	jessie-backports Debian repo, or: `deb http://osbpo.debian.net/debian jessie-mitaka-backports main` `deb http://osbpo.debian.net/debian jessie-mitaka-backports-nochange main`	Debian Jessie, Debian Stretch	Packages are in our reprepro, component 'openstack-mitaka-jessie'
Newton	14	Stretch Debian mirrors	Stretch
Ocata	15	`deb http://osbpo.debian.net/debian ocata main`	Stretch	Packages are in our reprepro, component 'openstack-ocata-stretch'
Pike	keystone 12 nova 16 neutron 11 glance 15 designate 5	`deb http://osbpo.debian.net/debian/ stretch-pike-backports main` `deb http://osbpo.debian.net/debian/ stretch-pike-backports-nochange main`	Stretch
Queens	17	`deb http://osbpo.debian.net/debian stretch-queens-backports main` `deb http://osbpo.debian.net/debian stretch-queens-backports-nochange main`	Stretch
Rocky	18	Official Debian mirrors for Buster. For Stretch backports: `deb http://osbpo.debian.net/debian stretch-rocky-backports main` `deb http://osbpo.debian.net/debian stretch-rocky-backports-nochange main`	Stretch, Buster
Stein	19	`deb http://osbpo.debian.net/debian buster-stein-backports main` `deb http://osbpo.debian.net/debian buster-stein-backports-nochange main`	Buster
Train	20	`deb http://osbpo.debian.net/debian buster-train-backports main` `deb http://osbpo.debian.net/debian buster-train-backports-nochange main`	Buster
Openstack release	Openstack version	DEB repository	Compatible Debian version	Notes

Server vs Client packages

We differentiate between clients and server packages.

clients are usually python libraries and utilities that interact with each component restful API.
server packages provide the daemons and tooling required to run the service itself (including the APIs).

Most of client software components are installed by means of Debian packages.

puppet code organization

Properly handling the version matrix in puppet code can be challenging. The layout we are trying to use right now is outlined here.

There are 2 main profiles that should be declared by each openstack deployment:

profile::openstack::base::clientpackages
profile::openstack::base::serverpackages

Usually, these profiles are just declared in the deployment-specific override, example:

profile::openstack::eqiad1::clientpackages --> declares --> profile::openstack::base::clientpackages
profile::openstack::eqiad1::serverpackages --> declares --> profile::openstack::base::serverpackages

This profile is responsible for calling the right openstack version/debian version module which contains the actual configuration, for example:

profile::openstack::base::clientpackages --> declares --> openstack::clientpackages::${version}::${::lsbdistcodename}
profile::openstack::base::serverpackages --> declares --> openstack::serverpackages::${version}::${::lsbdistcodename}

Here, ${version} is the openstack release in use, which is a different hiera key per deployment.

Each concrete module contains the specifics bits to be able to install the right packages in the system.

In case there are shared bits for client and server packages, these should be placed in the openstack::commonpackages::${version} class, and this clase should be declared require in the callers. Example:

openstack::serverpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka
openstack::clientpackages::mitaka::stretch --> require --> openstack::commonpackages::mitaka

Here is a summary list of things that can be included in each class:

profile::openstack::eqiad1::serverpackages call to the base profile, eqiad1 server package configuration override if required
profile::openstack::base::serverpackages call to the inner class, global server package configuration overrides if required
openstack::serverpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common server package installations
profile::openstack::eqiad1::clientpackages call to the base profile, eqiad1 client package configuration override if required
profile::openstack::base::clientpackages call to the inner class, global client package configuration overrides if required
openstack::clientpackages::${version}::${::lsbdistcodename} repo configuration, apt pinning and common client package installations
openstack::commonpackages::${version} stuff that is common to both of the above (factorization)

Then, How are component packages really installed? compontents are installed by its own profile/class path, that installs and configures it. Example:

profile::openstack::eqiad1::nova::compute::service
- profile::openstack::base::nova::compute::service
  - openstack::nova::compute::service
    - openstack::nova::compute::service::mitaka::stretch
      - (require) openstack::serverpackages::mitaka::stretch
        (require) openstack::commonpackages::mitaka

And, Is such level of indirection / abstraction really needed?
We had really hard times in the past dealing with the version matrix. We were forced to adopt this layout to be able to manage the puppet code for each component/openstack release/debian version sanely.
The level of indirections and abstraction is the necessary evil.

importing packages into our reprepro

In some cases, .deb packages aren't always available. Example is the archival of some Debian backports repository. This already happened, for example in T216497 - CloudVPS: workaround archival of jessie-backports repo.

A common and safe workaround is to import the packages into reprepro.

This is valid in cases such as:

we don't trust the availability of the original repository
the original repository is about to be deleted for some reason
a given repository doesn't contain all the openstack suite

In all these cases, it's advisable to collect and import all openstack packages into reprepro.

From source code

There are some components that are deployed/used directly from upstream source code, without Debian packages being involved. This is usually horizon.

TODO: fill me.

Portal:Cloud VPS/Admin/Openstack source