This page is outdate and refers to a setup we no longer have (shared keystone between deployments)

This page contains some notes on keystone, specifically how we deployed keystone for Cloud VPS.

previous and current status

Originally we had 1 keystone per deployment.

Then, Chase merged/collapsed the keystone services in labtest/labtestn, so a single keystone database serves both deployments. This means that the keystone database stores endpoints for both deployments, making use of the openstack concept of regions.

The reason for this merge is to allow as smooth migration from the nova-network to the neutron models.

On 2018-08-13 we merged also the main/eqiad1 keystone services (see phab:T201504).

endpoints in keystone

+----------------------------------+----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------+
| 28b4f3ada7984ee9aef1a40c38b52417 | eqiad1-r | glance       | image        | True    | admin     | http://cloudcontrol1003.wikimedia.org:9292                            |
| c71299aff8da47c7a03f3a2184d8b64b | eqiad1-r | glance       | image        | True    | internal  | http://cloudcontrol1003.wikimedia.org:9292                            |
| f922afd9417448028ce02734e0420a0b | eqiad1-r | glance       | image        | True    | public    | http://cloudcontrol1003.wikimedia.org:9292                            |
| 7c1fb33eb39f4eb19be510555cebd542 | eqiad1-r | keystone     | identity     | True    | admin     | http://cloudcontrol1003.wikimedia.org:35357/v3                        |
| fb991347d1d8409c9977ef0883e0917c | eqiad1-r | keystone     | identity     | True    | internal  | http://cloudcontrol1003.wikimedia.org:5000/v3                         |
| be7a84a1af114f94bd1d6cc48b374413 | eqiad1-r | keystone     | identity     | True    | public    | http://cloudcontrol1003.wikimedia.org:5000/v3                         |
| 30778a2f853843a3ae49a485919910fb | eqiad1-r | neutron      | network      | True    | admin     | http://cloudcontrol1003.wikimedia.org:9696                            |
| 4c1a383c90344c51978b6848c22836f9 | eqiad1-r | neutron      | network      | True    | internal  | http://cloudcontrol1003.wikimedia.org:9696                            |
| 0a1eb902933c4652ad41e6450fe436ee | eqiad1-r | neutron      | network      | True    | public    | http://cloudcontrol1003.wikimedia.org:9696                            |
| ce6257db56114db09ea14e55c62b73cd | eqiad1-r | nova         | compute      | True    | admin     | http://cloudcontrol1003.wikimedia.org:8774/v2.1                       |
| 1fe91d5e2ea344a18870a90e3abe310c | eqiad1-r | nova         | compute      | True    | internal  | http://cloudcontrol1003.wikimedia.org:8774/v2.1                       |
| 4578c49346db479ab6d5b7961af8f60a | eqiad1-r | nova         | compute      | True    | public    | http://cloudcontrol1003.wikimedia.org:8774/v2.1                       |
| 03ee231bc6334fd19edb32e9c6fa7465 | eqiad    | designate    | dns          | True    | admin     | http://labservices1001.wikimedia.org:9001                             |
| e1d0f85948f04bca9fd8d7628703a225 | eqiad    | designate    | dns          | True    | internal  | http://labservices1001.wikimedia.org:9001                             |
| b2621389830a4b29adb19886b8a28176 | eqiad    | designate    | dns          | True    | public    | http://labservices1001.wikimedia.org:9001                             |
| 655c239fd6e94e6981358811efd01b4e | eqiad    | glance       | image        | True    | admin     | http://labcontrol1001.wikimedia.org:9292                              |
| 1693cb5c47114161af07338765650cac | eqiad    | glance       | image        | True    | internal  | http://labcontrol1001.wikimedia.org:9292                              |
| a07a647fbb67452882793377ca9eab52 | eqiad    | glance       | image        | True    | public    | http://labcontrol1001.wikimedia.org:9292                              |
| 989fbfc7878244dab849d1c49051f2ba | eqiad    | keystone     | identity     | True    | admin     | http://cloudcontrol1003.wikimedia.org:35357/v3                        |
| e0ae0c5907f646fca23b13e73f2eb725 | eqiad    | keystone     | identity     | True    | internal  | http://cloudcontrol1003.wikimedia.org:5000/v3                         |
| 26038b5357844ec2816bbd6f06a8ee15 | eqiad    | keystone     | identity     | True    | public    | http://cloudcontrol1003.wikimedia.org:5000/v3                         |
| b966114d88334679b4d32f61c5dcfb31 | eqiad    | nova         | compute      | True    | admin     | http://labnet1001.eqiad.wmnet:8774/v2.1                               |
| 6fcd6be1f49d4d928ffedf44e6df0086 | eqiad    | nova         | compute      | True    | internal  | http://labnet1001.eqiad.wmnet:8774/v2.1                               |
| 4641c47238a14c68b45e831f7ec89b24 | eqiad    | nova         | compute      | True    | public    | http://labnet1001.eqiad.wmnet:8774/v2.1                               |
| 45f80e2d476b4408b388152e8edc1eac | eqiad    | proxy        | proxy        | True    | admin     | http://proxy-eqiad.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| 73b78e5e22004fb098ec7dca57b0b873 | eqiad    | proxy        | proxy        | True    | internal  | http://proxy-eqiad.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
| 32cf600161824ed5a46fd850d4bf4e27 | eqiad    | proxy        | proxy        | True    | public    | http://proxy-eqiad.wmflabs.org:5668/dynamicproxy-api/v1/$(tenant_id)s |
+----------------------------------+----------+--------------+--------------+---------+-----------+-----------------------------------------------------------------------+

Please note where the databases lives:

main/eqiad1: m5-master.eqiad.wmnet
labtest/labtestn: labtestcontrol2001.wikimedia.org

basic deployment

Please follow the bootstrap instructions modules/openstack/templates/bootstrap/keystone/keystone_seed.sh.erb

creating a shared keystone

Things to consider when collapsing keystone to server more than one deployment (example tracking phabricator task: phab:T201504)

networking connectivity (ferm ACLs) between the clients, the endpoints and the databases
regions in the database are correctly set
multi-region awareness. Lots of code using the keystone API isn't region-aware.
nova_controller vs keystone_host: we have 2 different hiera keys for setting where the keystone daemon lives. The latter is preferred.
endpoints are updated in the database to reflect the final environment

example

select * from endpoint where url LIKE "%labtestcontrol%";

UPDATE endpoint set \
 url='http://labtestcontrol2003.wikimedia.org:5000/v3' \
 WHERE url='http://labtestcontrol2001.wikimedia.org:5000/v3';

UPDATE endpoint set \
 url='http://labtestcontrol2003.wikimedia.org:35357/v3' \
 WHERE url='http://labtestcontrol2001.wikimedia.org:35357/v3';

keystone daemon only runs where it should run (the labcontrol box in the neutron deployment)
the new database should contain endpoints and services for both deployments being merged
related puppet patches:
- https://gerrit.wikimedia.org/r/#/c/433734/ openstack: labtest use labtestcontrol2003 for keystone
- https://gerrit.wikimedia.org/r/#/c/437783/ openstack: labtest: keystone: delete service (collapsed)
- https://gerrit.wikimedia.org/r/#/c/436853/ openstack: allow glance to call back for token validation
- https://gerrit.wikimedia.org/r/#/c/437812/ openstack: allow designate in labtest to contact labtestn keystone