Fundraising/techops/procedures/services-kafkatee

Services: Kafkatee/Kafka Client

Notes on the legacy Kafkatee banner impression pipeline.

Kafka Certificate

Kafka uses client certificate authentication for client connections. Kafka client certificates expire every six months, see https://phabricator.wikimedia.org/T360779. This covers how to obtain and deploy the certificate.

Kafka client certificates are generated automatically by production puppet ten days prior to expiration. We have icinga monitoring expiration, and will start getting warnings at this time. At eight days the alert goes to critical and we will start getting paged. There is also a systemd timer on cumin* that should email us.

The certificate can be retrieved from cumin\d{4}:/etc/fr-tech-kafka-client/*, there you'll find:

kafka__kafka_fundraising_client_kafka_11.chained.pem
kafka__kafka_fundraising_client_kafka_11.chain.pem
kafka__kafka_fundraising_client_kafka_11.csr
kafka__kafka_fundraising_client_kafka_11-key.pem
kafka__kafka_fundraising_client_kafka_11.pem

Copy these to (frack)puppet-private/secrets/kafkatee and do the git commit/push, and 'puppet-merge private' to deploy.

Update Cheatsheet

On cumin1003.eqiad.wmnet:

$ tar -cvf kafkatee_cert_20250508.tar /etc/fr-tech-kafka-client/
$ openssl enc -aes-256-cbc -pbkdf2 -in kafkatee_cert_20250508.tar -out kafkatee_cert_20250508.tar.enc
enter AES-256-CBC encryption password: <PASSWORD_GOES_HERE>
Verifying - enter AES-256-CBC encryption password: <PASSWORD_GOES_HERE>
$

On your localhost:

$ scp cumin1003.eqiad.wmnet:kafkatee_cert_20250508.tar.enc .
$ scp kafkatee_cert_20250508.tar.enc frpm1002:
$ shred -u kafkatee_cert_20250508.tar.enc

On frpm1002:

$ openssl enc -d -aes-256-cbc -pbkdf2 -in kafkatee_cert_20250508.tar.enc -out kafkatee_cert_20250508.tar
enter AES-256-CBC decryption password: <PASSWORD_GOES_HERE>
$ tar -xvf kafkatee_cert_20250508.tar
$ cp ~/etc/fr-tech-kafka-client/*.pem ~/puppet-private/secrets/kafkatee/
$ cd ~/puppet-private/secrets/kafkatee/
$ git commit *.pem