This page summarizes the learnings of https://phabricator.wikimedia.org/T365005 . That phab task explored an alternative architecture design to refactor stream configuration services (Stream Registry) to better integrate with a platform wide Config Store and MPIC use cases.

After re-scoping both Config Store and MPIC, we decided to not move forward with refactoring ESC at this stage. A Design Document is available at Stream Registry

Stream Registry

2024-05-17

URL:Stream Registry

Author: Gabriele Modena <gmodena@wikimedia.org>

Bug: T365005

This is a living document, and currently (as of 2024-05-23) under review.

Overview

This document describes the design of a Stream configuration specification and its reference implementation as a web service (Stream Registry), deployed on Kubernetes (multi data center, targeting wikikube). Stream Registry will be initially backed by Git and Cassandra, with the goal of moving to an s3 protocol compatible object store once CEPH is available. Stream Registry will not need to be directly exposed to the public internet, only the internal network or a selected pool of IPs.