Release Engineering/SAL

2026-05-15

13:02 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1287828 T426392

2026-05-13

12:42 James_F: Zuul: [mediawiki/extensions/Springboard] Add AdminLinks Phan dependency
12:42 James_F: Zuul: [mediawiki/extensions/ChatBot] Add dependencies on VisualEditor and BlueSpiceFoundation
12:42 James_F: Zuul: [mediawiki/extensions/ChatIntegration] Add dependency on VisualEditor
12:37 James_F: Zuul: [mediawiki/extensions/WikiLambda] Drop AF and SB deps down to phan-only, for T423180

2026-05-12

20:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/104 (T424774)
18:08 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add AF and SB deps for T423180
14:18 atsukoito: PrivateSettings: empty $wgOpensearchCredentials for opensearch-on-k8s synced to deploy04 by Reedy
13:04 atsukoito: PrivateSettings: credentials for opensearch-on-k8s ttmserver-test
11:50 James_F: Zuul: [machinelearning/liftwing/inference-services] Add qwen36 llm model CI/CD pipelines, for T425680
11:46 James_F: Zuul: Add experimental php-pie-build* jobs to other PHP extensions, for T425943
11:37 James_F: Zuul: [mediawiki/php/wikidiff2] Add experimental php-pie-build* jobs, for T425943
10:05 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83 # fix failure seen in quibble-with-Wikibase-extensions-browser-tests-only-vendor-php83 7817
08:44 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/ # broken Cypress cache? hopefully fix failure seen in quibble-vendor-mysql-php83-selenium 51633

2026-05-11

18:28 James_F: Docker: Add changes to php-compile images for PIE, for T425943
16:06 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/ # broken Cypress cache? hopefully fix failure seen in quibble-vendor-mysql-php83-selenium 51439 and 51452

2026-05-09

20:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1285498

2026-05-07

22:53 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/105

2026-05-06

18:13 bd808: Unblock 88.165.192.0/19
18:03 bd808: Unblock 94.208.0.0/14
17:56 bd808: Unblock 84.226.0.0/16
17:41 bd808: Unblock 94.34.0.0/16
17:35 bd808: Unblock 109.134.0.0/16

2026-05-05

21:20 James_F: Zuul: Provide Node 26 experimental jobs everywhere needed
21:04 James_F: Docker: Provide initial Node 26 images
19:01 James_F: Zuul: [mediawiki/extensions/PageAssessments] Add Scribunto dependency, for T396135
14:58 dancy: rm /var/log/{user.log.1,syslog.1,messages.1} on deployment-eventgate-4.deployment- prep (T425429)

2026-05-04

15:19 dancy: Upgrading gitlab cloud runners (prod) from 1.35.1-do.3 to 1.35.1-do.5
14:51 dancy: Upgrading gitlab cloud runners (staging) from 1.35.1-do.3 to 1.35.1-do.5
10:40 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for bluespice template

2026-05-02

20:49 James_F: Zuul: [mediawiki/core] Enforce PHP 8.4 & 8.5 on release branches, all pass
19:27 James_F: Zuul: Provide non-voting PHP 8.4/8.5 Quibble jobs for MW release branches
19:19 James_F: Zuul: [mediawiki/extensions/BlogPage] Add dependencies
16:48 James_F: Hard-restarting Zuul to clear the huge number of i18n updates being re-submitted.
15:48 James_F: Zuul: [wikimedia-cz/*] Test in PHP 8.3+, dropping 8.2
14:02 TheresNoTime: Add bvibber to deployment-prep project
09:08 James_F: Docker: [quibble-*] Add php-luasandbox so we can test both modes in Scribunto

2026-05-01

15:42 James_F: Zuul: [wikimedia/lucene-explain-parser] Test in PHP 8.3+, dropping 8.2
15:42 James_F: Zuul: [wikimedia/textcat] Test in PHP 8.3+, dropping 8.2
15:42 James_F: Zuul: [mediawiki/tools/ParseWiki] Test in PHP 8.3+, dropping 8.2
15:42 James_F: zuul: Add ToprakM to CI allowlist
15:19 James_F: Zuul: [translatewiki] Test in PHP 8.3+, dropping 8.2
15:10 James_F: Zuul: [mediawiki/extensions/WikiEditor] Add TestKitchen as a dependency, for T425076
12:40 James_F: Zuul: [mediawiki/tools/code-utils] Test in PHP 8.3+, dropping 8.2
08:02 James_F: Zuul: Update xtex's e-mail in the allowlist
07:37 James_F: Zuul: Switch release branches' selenium jobs to PHP 8.3
07:33 James_F: Zuul: Test Wikimedia production libraries in PHP 8.3+, dropping 8.2

2026-04-30

21:36 brennen: gitlab-webhooks: building & restarting to deploy https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/40
20:26 James_F: Zuul: [mediawiki/tools/api-testing] Make PHP 8.5 CI voting
20:16 James_F: jforrester@doc1004:~$ # sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn/ # T415832
20:14 James_F: Zuul: [mediawiki/extensions/WebAuthn] Archive, for T415832 / T303495
17:16 brennen: wikibugs: most maintainers at hackathon, so go release-engineering added as a maintainer while looking to debug error at https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/jobs/810904
15:19 mutante: upgrading zuul to 14.2.0-1 on "new zuul" machines (T424879)

2026-04-29

15:49 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add ConfirmEdit dependency, for T424597
15:36 James_F: Zuul: Drop experimental node22 jobs, never used in practice
15:28 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1279392, https://gerrit.wikimedia.org/r/1279397

2026-04-28

18:11 bd808: Unblock 86.0.0.0/16
17:41 bd808: Unblock 79.192.0.0/10
17:07 James_F: Zuul: [mediawiki/tools/phpunit-patch-coverage] Drop PHP 8.2 testing
17:07 James_F: Zuul: [mediawiki/tools/minus-x] Drop PHP 8.2 testing
17:07 James_F: Zuul: [mediawiki/tools/codesniffer] Drop PHP 8.2 testing
16:32 James_F: Zuul: [mediawiki/services/jobrunner] Drop PHP 8.2 testing
13:34 James_F: Zuul: [mediawiki/tools/phan] Drop PHP 8.2 testing
13:34 James_F: Zuul: [oojs/ui] Drop PHP 8.2 testing
13:14 James_F: Zuul: [mediawiki/tools/phan/SecurityCheckPlugin] Drop PHP 8.2 CI
10:40 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-rundoc/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud to fix failure seen in mwext-node24-rundoc #1717
00:03 bd808: Increase parallelism for wmf-beta-update-databases.py (T256168)

2026-04-27

22:11 bd808: Beta Cluster MediaWiki update logs now available via https://beta-update.wmcloud.org/ (T256168)
21:57 bd808: Add web security group to deployment-deploy04 (T256168)
20:45 James_F: Zuul: Restrict mw*-codehealth-patch jobs to master only, for T424573
17:16 James_F: Docker: [mediawiki-phan-taint-check-demo] Re-platform to Trixie and so PHP 8.4
15:53 James_F: Zuul: [mediawiki/extensions/ReportIncident] Add TestKitchen phan dependency, for T424220
14:32 James_F: Zuul: Drop PHP 8.2 enforcement from MediaWiki things for master and REL1_46 for T358667
12:38 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mwext-node24-docs-publish # fix failure seen in mwext-node24-docs-publish 383
09:18 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # T424419

2026-04-26

20:49 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276777

2026-04-24

22:48 dduvall: merged zuul3 branch of integration/config into master and pushed (in preparation for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1277198)
12:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1276428

2026-04-23

23:57 bd808: Set `profile::beta::autoupdater::run_updater: true` for deployment-deploy04 via Horizon (T256168)
22:58 bd808: bd808@deployment-deploy04 `sudo -u jenkins-deploy /usr/local/bin/wmf-beta-update-all`
22:36 bd808: bd808@deployment-deploy04 `sudo -u mwdeploy /usr/local/bin/wmf-beta-update-all`
22:16 bd808: Disabled https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad so that replacement script can be tested (T256168)
22:12 bd808: Disabled https://integration.wikimedia.org/ci/job/beta-code-update-eqiad so that replacement script can be tested (T256168)
22:02 bd808: Cherry-picked 1276813 to deployment-puppetserver-1 (T256168)
20:11 James_F: Zuul: [wikibase/*] Replace CI testing in Node 20 with Node 24
20:11 James_F: Zuul: [wikidata/query/*] Replace CI testing in Node 20 with Node 24
20:11 James_F: Zuul: [analytics/*] Replace CI testing in Node 20 with Node 24
20:10 James_F: Zuul: [mediawiki/tools/*] Replace CI testing in Node 20 with Node 24
20:06 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.34.5-do.3 to 1.35.1-do.3 (T423726)
19:55 James_F: Zuul: [jquery-client] Replace CI testing in Node 20 with Node 24
19:51 James_F: Zuul: [wikipeg] Drop testing in Node 20 and Node 22
19:47 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.34.5-do.3 to 1.35.1-do.3 (T423726)
19:37 James_F: Zuul: [oojs/ui] Drop CI testing in Node 20 and Node 22
19:37 James_F: Zuul: [oojs/js] Drop CI testing in Node 20 and Node 22
19:37 James_F: Zuul: [unicodejs] Replace CI testing in Node 20 with Node 24
19:36 James_F: Zuul: [wikimedia/portals] Drop CI testing in Node 20 and Node 22
18:57 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.33.9-do.3 to 1.34.5-do.3 (T423726)
18:39 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.3 to 1.34.5-do.3 (T423726)
18:18 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.33.9-do.2 to 1.33.9-do.3 (T423726)
17:58 James_F: Zuul: [mediawiki/extensions/OAuth] Add dependency on CentralAuth, for T415281
17:56 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.13-do.2 to 1.33.9-do.3 (T423726)
16:35 James_F: Zuul: Enforce PHP 8.5 CI for MW things in master (and REL1_46), for T411814
16:19 James_F: Zuul: [mediawiki/services/parsoid] Enable PHP 8.5 CI
15:47 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add AntiSpoof dependency, for T420548
14:20 Lucas_WMDE: ssh integration-castor06.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24 # fix failure seen in mediawiki-node24 8385 and 8405
12:56 James_F: Zuul: [mediawiki/extensions/GrowthExperiments] Add CentralNotice dependency, for T422082

2026-04-22

00:07 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add MF dependency, for T424113

2026-04-21

23:26 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration dep too, for T394410
23:17 James_F: Zuul: [mediawiki/extensions/DiscussionTools] Add standalone test jobs, for T422031
20:47 inflatador: updating cirrussearch hosts to Trixie/OpenSearch 2 T421763
20:38 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add CommunityConfiguration phan dep, for T394410
20:17 bd808: Running tofu for T421244
18:00 James_F: Zuul: [mediawiki/extensions/WatchAnalytics] Add ApprovedRevs Phan dependency
16:35 bd808: Unblock 79.116.0.0/16
13:34 James_F: Zuul: [mediawiki/extensions/WikiLambda] Add TestKitchen phan dep, for T415254
13:27 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add CentralAuth dependency, for T420548

2026-04-20

23:56 bd808: Unblock 76.157.0.0/16
18:28 dancy: Upgrading gitlab cloud runners (staging) to 1.33.9-do.2 (T423726)
18:28 dancy: Upgrading gitlab cloud runners (staging) (T423726)
18:19 James_F: jjb: All 486 (!) jobs now updated for T423622
18:18 bd808: Unblock 113.128.0.0/15
15:03 James_F: Docker: Bump ci-bullseye/-bookworm/-trixie for mirrors.wm.org removal, T423622

2026-04-19

19:53 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272752

2026-04-17

21:07 thcipriani: marking integration-agent-1080 offline for experimentation
19:30 thcipriani: reconfiguring castor-save-workspace-cache with https://gerrit.wikimedia.org/r/1273935
17:47 dancy: Upgrading gitlab cloud runners (prod) k8s from 1.32.10-do.1 to 1.32.13-do.2 (T423726)
16:49 dancy: Upgrading gitlab cloud runners (staging) k8s from 1.32.10-do.1 to 1.32.13-do.2 (T423726)

2026-04-16

20:49 dduvall: creating integration/zuul-jobs repo to serve as a mirror of opendev.org/zuul/zuul-jobs (T406384)
13:38 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1272711 T423568
11:07 Silvan_WMDE: sudo -u jenkins-deploy rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node24/ # run on integration-castor06.integration.eqiad1.wikimedia.cloud

2026-04-15

20:05 James_F: Zuul: Configure REL1_46 CI, for T423257
17:44 bd808: Unblock 176.0.0.0/13
17:39 bd808: Unblock 46.128.0.0/16
17:32 bd808: Unblock 176.86.0.0/16
16:39 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/commit/127d783b2176ac60b646a5fa4f1b1a872ca66340
15:33 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/100
01:02 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/99

2026-04-14

20:42 James_F: Docker: [composer-scratch] Upgrade composer to 2.9.7 and cascade
16:35 bd808: Unblock 88.112.0.0/14
00:48 bd808: Unblock 24.6.0.0/16
00:42 bd808: Unblock 152.231.48.0/20

2026-04-13

22:00 James_F: Zuul: [mediawiki/vendor] Drop accidental Wikibase browser tests on branches
20:28 James_F: Zuul: [mediawiki/extensions/Chart] Drop Doxygen publish job, not used
14:42 James_F: Zuul: [mediawiki/extensions/WikimediaCustomizations] Add FlaggedRevs dep, for T421011

2026-04-12

18:21 James_F: jforrester@contint1002:~$ sudo /usr/sbin/service zuul restart && tail -f -n100 /var/log/zuul/zuul.log # T423027

2026-04-10

23:22 James_F: jforrester@contint1002:~$ zuul enqueue --trigger gerrit --pipeline postmerge --project mediawiki/extensions/ReadingLists --change 1269498,2 # T422976
23:20 James_F: Zuul: [mediawiki/extensions/ReadingLists] Publish JS coverage, for T422976
23:13 James_F: Zuul: Migrate a few straggler Node 20 MediaWiki things to Node 24
23:01 James_F: Zuul: Move all MediaWiki things from mediawiki-node20 to mediawiki-node24
21:59 James_F: Docker: Bump Node base images to March releases and cascade; Upgrade Quibble images from Node 20 to Node 24
10:24 hashar: Updating all Quibble jobs to 1.17.1
10:22 hashar: Updated PostgreSQL jobs to Quibble 1.17.1 # T422110
10:22 hashar: Updated apitesting job to Quibble 1.17.1 # T422843 T418743
09:51 hashar: Tag Quibble 1.17.1 @ 0a1ab3b

2026-04-09

15:13 hashar: Rolling back Quibble jobs to 1.16.0 (api-testing stage fails due to missing npm install step`
14:58 hashar: Upgrading Quibble jobs to 1.17.0
14:23 hashar: Tagged Quibble 1.17.0 @ 864381c
07:48 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1268559 "Zuul: use standalone jobs for GrowthExperiments Cypress tests" | T417412

2026-04-08

22:19 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1269068
22:01 bd808: Unblock 95.216.12.170/32 (T422751)
19:26 brennen: gitlab-webhooks: building & deploying https://gitlab.wikimedia.org/repos/releng/gitlab-webhooks/-/merge_requests/37 - hitting some build tooling stuff, trying a fix per instructions in the error log
17:54 bd808: Unblock 167.56.0.0/13 (T422721)
06:31 hashar: Deleted integration-agent-castor05 Bullseye instance, replaced by integration-agent-castor06 which is on Bookworm # T421114
06:24 hashar: Deleted integration-agent-qemu-1003 Bullseye image, replaced by integration-agent-qemu-1004 which is on Bookworm # T422488

2026-04-07

22:25 dduvall: adding new pipelinelib labels to ci nodes (T422234)
20:05 hashar: Triggered a build of https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/
17:06 dduvall: added `Docker` label to `contint` jenkins nodes (T422507)
17:05 dduvall: restored missing `pipelinelib` labels on `integration-agent-docker-` CI hosts (T422507)
16:53 bd808: Unblock 73.0.0.0/8 (T422498)
12:36 hashar: jjb: use $CASTOR_HOST for Quibble success cache. https://gerrit.wikimedia.org/r/1268545 | This causes the Quibble jobs to use a new instance for the success cache, which is empty # T383243 T421114
12:17 hashar: Migrated Castor from integration-castor05 to integration-castor06. Updated CASTOR_HOST in Jenkins and moved the Cinder volume to the new instance # T421114
11:14 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1090, 1091, 1092 and 1093 # T421114
10:09 hashar: Added Bookworm based Jenkins agents to the pool Hostnames 1083 to 1089 # T421114
07:23 hashar: CI Jenkins: removed `blubber` label from all agents after having moved PipelineLib to use the `Docker` label | T422234

2026-04-06

16:01 dancy: Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/c/integration/config/+/1268239

2026-04-03

20:17 bd808: Unblock 2.54.0.0/16 (T422238)
17:25 bd808: Unblock 31.18.0.0/16 (T422245)
17:18 bd808: Unblock 2.54.128.0/19 (T422238)
16:18 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1264649 "add Python 3.14 to pywikibot jobs and separate lint tests" | T421723
09:26 hashar: integration: nuked pywikibot/core pre-commit cache # T422242
09:15 hashar: Added Bookworm based Jenkins agents to the pool with label `Docker`. Hostnames are `integration-agent-docker-107*` # T421114
02:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1267398

2026-04-02

16:50 thcipriani: restart jenkins
15:15 bd808: Unblock 82.216.0.0/16 (T421508)
15:07 bd808: Unblock 95.90.0.0/15 (T421485)
11:19 James_F: Zuul: [oojs/ui] Drop ooui-ruby2.7-rake job, we're abandoning Ruby use there

2026-04-01

22:01 bd808: Unblock 109.144.0.0/12 (T422019)
20:16 bd808: Unblock 93.192.0.0/10 (T421894)
19:25 dancy: Updating buildkitd to v0.29.0 in gitlab-cloud-runners (prod) (T415284)
17:57 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/97 (T420441)
17:39 bd808: Unblock 94.134.0.0/15 (T421866)
16:31 dancy: Upgrade buildkit to 0.29.0 in staging gitlab-cloud-runners (T415284)
10:47 taavi: integration-castor05: free up a bit of disk space by deleting cache for AhoCorasick/ CLDRPluralRuleParser/ HtmlFormatter/ RelPath/ RunningStat/ IPSet/

2026-03-30

22:01 bd808: Unblock 78.20.0.0/14 (T421586)
21:04 bd808: Unblock 95.88.0.0/15 (T421774)
20:49 bd808: Unblock 95.89.191.0/24 (T421774)
20:29 bd808: Unblock 73.162.0.0/16 (T421549)
13:10 hashar: gerrit: abandon mediawiki/core changes that are 2+years old and are attached to a task (`Bug: Txxxx`)
11:37 hashar: Reloaded Zuul to to add 3 persons to the allow list
10:43 James_F: Docker: Re-pushing to try to create quibble-coverage 1.16.0-s2

2026-03-27

21:00 James_F: Docker: [quibble-bullseye] Drop Python 2 from images
11:28 hashar: deployment-prep: removed block for `143.176.0.0/15` and blocked subblock `143.176.0.0/16` instead. This unblocks `143.177.0.0/16` # T421420
00:18 bd808: Unblock 95.90.238.0/23 (T421447)

2026-03-26

21:25 bd808: Unblock 89.240.0.0/15 (T421364)
21:09 brennen: patchdemo: deploy to production for https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/312

2026-03-25

20:41 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256318 T421283
15:46 dancy: Migrated gitlab-cloud-runners (prod) from nginx-ingress to traefik (T420743)
15:32 dancy: Migrated gitlab-cloud-runners (staging) from nginx-ingress to traefik (T420743)
10:01 hashar: Updating tox Jenkins jobs to add support for Python 3.14 | https://gerrit.wikimedia.org/r/1260632 | T421209
08:40 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20/

2026-03-24

19:40 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255746
15:34 brennen: gitlab1004: manual test run of `configure-projects` with cleared issue allowlist (T412882)
15:26 bd808: Unblock 47.194.0.0/16 (T421127)
12:53 hashar: integration: deleted old Puppet 5 compiler agents from Jenkins ( pcc-worker1014.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1015.puppet-diffs.eqiad1.wikimedia.cloud , pcc-worker1016.puppet-diffs.eqiad1.wikimedia.cloud ) # T367399
07:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1259755

2026-03-23

15:28 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 90272

2026-03-22

14:52 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1258082
01:00 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256488

2026-03-21

08:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256962
07:48 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1256946

2026-03-20

21:21 bd808: Unblock 103.159.218.0/24 (T420530)
14:59 James_F: Zuul: [mediawiki/extensions/AbuseFilter] Add dependency on CodeMirror, for T399673

2026-03-19

16:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1255777
16:01 Krinkle: Hoist l10n-bot rights from labs/tools parent to labs parent to reduce duplication in other labs/ repos
15:50 Krinkle: Create labs/xtools repo (branch: main, parent: labs, owner: labs-xtools), ref T402086

2026-03-18

21:11 dcausse: T403775: reindexing all wikis to enable new sorting options
21:08 dcausse: restarting opensearch on deployment-cirrussearch(12|13|14) instances to pickup new plugin versions
14:56 James_F: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest
14:52 James_F: Zuul: [GrowthExperiments] drop duplicate VisualEditor dep
14:52 James_F: Zuul: [search/*] Add experimental Java 25 jobs

2026-03-17

22:50 James_F: Zuul: [mediawiki/extensions/JsonForms] Add quibble jobs
21:27 James_F: Zuul: search: Update opensearch plugins for Java 11/17, for T420407
20:20 bd808: Resize deployment-sessionstore06 from g4.cores1.ram2.disk20 to g4.cores2.ram4.disk20 (T415021)
16:43 James_F: Zuul: [BlueSpicePermissionManager] Add …ConfigManager & …UserManager deps
14:36 James_F: Zuul: [mediawiki/extensions/ArticleGuidance]: Add SpamBlacklist as phan dep, for T420015

2026-03-13

13:59 andrewbogott: deleting ptr record 117.0.16.172.in-addr.arpa. -- accidental duplicate for deployment-kafka-logging01.deployment-prep.eqiad1.wikimedia.cloud
13:04 elukey: re-create kafka-logging-01 in deployment-prep on trixie and Kafka 3.7 (was running on buster)
09:13 elukey: upgrade kafka-jumbo and kafka-main to Confluent 7.7 in deployment-prep (pre-requisite before being able to upgrade to Trixie)

2026-03-12

21:23 bd808: Hard reboot deployment-sessionstore06 (T415021)
01:14 James_F: Docker: [helm-linter] Bump for Envoy 1.35.9, for T419637

2026-03-11

16:48 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/MetricsPlatform # T417568
16:47 James_F: Zuul: [mediawiki/extensions/MetricsPlatform] Archive, for T416865
11:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1250529 "inference-services: Split policy violation CI into separate model jobs." - T418832

2026-03-10

17:39 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner production
17:11 hashar: Updated MediaWiki coverage jobs so that they now keep "Generate a local configuration by running `composer phpunit:config`" message # T419073
16:41 dduvall: deployed reggie v1.18.0 to gitlab-cloud-runner staging
08:21 codders: integration: integration-castor05: rm -fR /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20

2026-03-09

21:53 bd808: Reboot deployment-shellbox01 on the off chance that is makes the new permissions error go away (T419440)
13:13 James_F: Zuul: [mediawiki/extensions/WikiShare] Mark as archived, for T413589
13:11 James_F: Zuul: [mediawiki/extensions/Memento] Mark as archived, for T369991
13:10 James_F: Zuul: [mediawiki/extensions/QuickGV] Mark as archived, for T413348
13:10 James_F: Zuul: [mediawiki/extensions/SemanticImageInput] Mark as archived, for T413588
13:09 James_F: Zuul: [mediawiki/extensions/SidebarDonateBox] Mark as archived, for T413587
13:07 James_F: Zuul: [mediawiki/extensions/SemanticSifter] Mark as archived, for T413586
13:06 James_F: Zuul: [mediawiki/extensions/GoogleAdSense] Mark as archived, for T413585
13:04 James_F: Zuul: [mediawiki/extensions/SecurityAPI] Mark as archived, for T418008
12:50 James_F: Zuul: [mediawiki/extensions/CheckUser] Add DiscussionTools dependency
12:50 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add dependencies for TestKitchen
10:40 hashar: gerrit: mediawiki/vendor: converted `es6` and `es710` branches to tags # T417804
09:24 hashar: Updating Quibble jobs to 1.16.0 | https://gerrit.wikimedia.org/r/c/integration/config/+/1248880 | T417399 T417409 T418461
09:15 hashar: updating all CI Jenkins jobs using `./jjb-update`

2026-03-06

19:46 James_F: Zuul: [mediawiki/services/geoshapes] Mark as archived, for T418372
16:37 hashar: Building Docker images for Quibble 1.16.0
16:31 hashar: Tag Quibble 1.16.0 @ 0b9db5f # T417399 T417409 T418461
12:32 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248411 "jjb, Zuul: vary Wikibase Selenium for release branches" | T418797
12:12 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1248409/ "jjb, Zuul: rename wikibase-selenium job for clarity" | T418797

2026-03-05

14:41 James_F: Zuul: [mediawiki/skins/MinervaNeue] Add TestKitchen as a dependency for T418053
08:01 hashar: Reloaded Zuul to rename wikibase-client / wikibase-repo jobs | https://gerrit.wikimedia.org/r/1238317
00:04 James_F: Docker: [quibble-coverage] Use local PHPUnit config, for T345481

2026-03-04

21:16 James_F: Zuul: [mediawiki/core] Make PHP 8.5 voting on master branch, for T411814
21:10 James_F: Zuul: [mediawiki/vendor] Make PHP 8.5 voting on master branch, for T411814
19:48 brennen: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/96 (T419004)
18:50 James_F: Revert "Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency", for T419043
16:23 James_F: Zuul: [mediawiki/services/parsoid] Make PHP 8.4 voting
15:37 James_F: Docker: [rake-ruby2.7] Add libffi-dev too, for T418463
13:59 James_F: Docker: [rake-ruby2.7] Add ruby-ffi for T418463
13:54 hashar: SIGKILL Zuul cause it can't gracefully stop most probably due to being locked attempting to report back to Gerrit # T419009
13:49 hashar: Stopping Zuul # T419009
13:41 hashar: Took a Zuul stack dump on contint1002.wikimedia.org using SIGUSR1 # T419009

2026-03-03

23:52 James_F: Zuul: [mediawiki/extensions/WikimediaMessages] Drop MetricsPlatform phan dep
23:52 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Drop MetricsPlatform phan dep

2026-03-02

22:13 James_F: Zuul: Enforce PHP 8.4 in MW extensions and skins for development branch, for T386108
14:05 James_F: Zuul: [mediawiki/extensions/MobileFrontend] Add ParserMigration dependency, for T415451
13:48 James_F: Zuul: […/WikimediaEvents] Drop LoginNotify dependency, now unused, for T404334
10:16 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/quibble-vendor-mysql-php83-selenium/Cypress/15.8.2/ # T418718

2026-02-28

21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/skins` # T418675
21:33 hashar: gerrit: triggering replication to GitHub for all of `mediawiki/extensions` # T418675

2026-02-27

15:53 dancy: Updating gitlab-cloud-runners (staging and prod) to gitlab-runner 18.9.0.

2026-02-26

20:16 James_F: Zuul: Provide a custom, high-priority pipeline just for puppet compiler T414621
19:32 James_F: Docker: Bump all the PHPs.
13:40 hashar: Deployed Jenkins job https://integration.wikimedia.org/ci/job/wikibase-selenium/ # T287582
00:13 dduvall: forcing replacement of buildkitd helm release in gitlab-cloud-runner prod cluster due to dependency on removed k8s secret (T416260)

2026-02-25

23:50 dduvall: deploying https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/552 to gitlab-cloud-runner production cluster (T416260)
14:07 James_F: Zuul: [mediawiki/extensions/CommunityRequests] Add TemplateData dependency, for T401638
00:08 jeena: no-op testing updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/95

2026-02-24

15:55 brennen: devtools: test deploy phab/phorge to test instance (T418256)

2026-02-23

23:07 jeena: Updated development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92
22:43 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/92
22:12 bd808: Unblock 191.80.192.0/18 (T418132)
20:26 hashar: Deleted "replication-upstream" Grafana dashboard in favor of a copy/new "replication" one. https://grafana.wikimedia.org/d/RFLS1GsWk/replication-upstream , replaced it by https://grafana.wikimedia.org/d/d4a4da73-c27f-4ce6-a9e5-ab84dd7a4ebb/replication
16:29 James_F: Zuul: [3d2png] Add basic Node CI at version 20

2026-02-20

21:47 bd808: Unblock 168.184.84.0/24 (T418020)
17:13 bd808: Unblock 122.187.64.0/18 (T417964)
14:35 James_F: Zuul: [mediawiki/extensions/Monstranto] Move out of Wikimedia prod section

2026-02-19

18:34 bd808: Unblock 181.98.0.0/16 (T417890)
17:21 James_F: Zuul: [mediawiki/extensions/WikimediaEvents] Add AbuseFilter as a dependency, for T417799
13:22 hashar: Reloaded Zuul to archive the Cergen repository | https://gerrit.wikimedia.org/r/c/integration/config/+/1240688 | T417887

2026-02-18

20:17 jeena: Updating development images on contint primary for T415922
19:44 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240360
18:40 bd808: Unblock 46.59.0.0/17 (T417747)
17:05 hashar: Regenerating Jenkins jobs with JJB based on https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/
17:04 hashar: Added EXT_DEPENDENCIES to Quibble Jenkins jobs parameters so we can manually trigger them from the Web UI using a different set of deps # https://gerrit.wikimedia.org/r/c/integration/config/+/1240254/
16:30 hashar: Triggered https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/ with empty Zuul parameters introduced by https://gerrit.wikimedia.org/r/1240333 | https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/4893/console
15:43 James_F: Zuul: [mediawiki/extensions/ReadingLists] Add EventBus dependency for T417706
12:15 hashar: zuul-1001.zuul3.eqiad1.wikimedia.cloud: added keepalive=20 to the scheduler Gerrit driver and restarted scheduler container # T417497
06:58 jeena: Updating development images on contint primary for T415922

2026-02-17

23:37 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240081
23:20 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1240078
15:58 brennen: deployed latest phab/phorge wmf/stable to devtools test instance (T417657)
09:01 hashar: Reloaded Zuul to enable php 8.5 testing on utfnormal, php-session-serializer, wikipeg, mediawiki/libs/Dodo, mediawiki/libs/UUID, testing-access-wrapper and translatewiki # T406326

2026-02-16

15:27 hashar: Manually cleaned some old workspaces on integration-agent-docker-1042

2026-02-12

20:07 James_F: Zuul: Enable PHP 8.5 jobs for most MW libraries, for T406326
19:33 James_F: Docker: [php83] Re-build with upstream's new 8.3.30 release and cascade
19:31 James_F: Zuul: Add PHP 8.5 CI job to various things noted as blocked by Phan, for T410941, T406326
16:35 Krinkle: Disable publishing noise on tasks from repos Bcp47, clover-diff, ScopedCallback, and IDLeDOM. Ref T143162
15:53 dancy: Updating development images on contint primary for https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/87
11:21 James_F: Zuul: [mediawiki/libs/shellbox] Add direct Phan job, for T416064

2026-02-10

20:16 dancy: Rebooted k3s.catalyst-dev (it was unresponsive, but the reboot hasn't helped)

2026-02-09

21:58 James_F: Zuul: [mediawiki/tools/phan] Add PHP 8.5 CI job, for T410941
19:46 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1238006 T415680
11:51 James_F: Zuul: [mediawiki/extensions/ReadingLists] Drop MetricsPlatform dependency, for T414435

2026-02-05

17:58 James_F: Zuul: […/WikimediaCustomizations] Add six new dependencies for T404334
15:35 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1237254
15:18 James_F: Zuul: […/OATHAuth] Add dependency and phan dependency on CentralAuth

2026-02-04

12:54 James_F: Zuul: [mediawiki/extensions/Petition] Add CLDR dependency
10:03 hashar: Restarted Jenkins on releases2003.codfw.wmnet

2026-02-02

21:17 hashar: Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/1234926 "re-enable master jobs for some BlueSpice repos - T403196"
21:05 bd808: Unblock 85.146.0.0/17 (T416079)
19:47 James_F: Zuul: […/WikimediaCustomizations] Add cldr phan dependency, for T404334
17:33 bd808: Unblock 188.188.0.0/15 (T416095)
17:26 bd808: Unblock 85.94.84.0/22 (T416105)
17:09 bd808: Unblock 94.234.0.0/16 (T416165)
16:51 dancy: Update gitlab-runners to alpine-v18.6.6 (T415214)
16:27 bd808: Unblock 47.231.208.0/21 (T416010)
11:39 James_F: Zuul: […/WikimediaCustomizations] Add five new phan dependencies, for T404334
09:45 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 58532, 58557

2026-01-31

21:49 James_F: Deleted Jenkins's job entry for castor-save-workspace-cache 6193776 and this seems to have unstuck things for T416078?
21:45 James_F: Running `sudo systemctl restart jenkins` on contint for T416078
21:44 James_F: Fighting T416078, took integration-castor-5 offline, disconnected, sshed in to kill threads, then reconnected; no change in aspect.
19:03 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1235380

2026-01-28

21:26 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/WebAuthn # T415832
21:11 bd808: Unblock 181.160.0.0/15 & 186.40.128.0/17 (T415820)
17:01 bd808: Unblock 102.182.0.0/16 (T415782)

2026-01-27

16:45 James_F: Zuul: Switch skin-quibble template with identical extension-quibble, for T402398
16:18 James_F: Zuul: [ArticleGuidance] mention it will be in production
15:55 James_F: Docker: [quibble-bullseye] Update to Quibble 1.15.0
15:12 James_F: Docker: [quibble-coverage] Pass PHPUnit config location explicitly, for T395470
09:18 hashar: integration: on integration-castor05, deleted caches for old MediaWiki branches
09:15 hashar: integration: on pkgbuilder instances, removed Buster cow images, aptcache and hooks. `sudo cumin --force -p 0 'name:pkgbuilder' 'rm -fR /srv/pbuilder/{base-buster-amd64.cow,hooks/buster,aptcache/buster-amd64}'` # T397209
09:14 hashar: integration: cleaned up old workspaces under /srv/jenkins/workspace

2026-01-26

23:27 bd808: Unblock 66.130.0.0/15 (T415596)
22:52 bd808: Unblock 45.16.0.0/12 (T415467)
14:46 hashar: gerrit: changed `operations/software/permissions` project type from `CODE` to `PERMISSIONS` by pointing `HEAD` to `refs/meta/config`

2026-01-22

17:36 James_F: Docker: [quibble-coverage] Stop using legacy PHPUnit entrypoint (T395470) & Stop excluding Dump/ParserFuzz/Stub groups (T415230)
15:11 James_F: Zuul: [mediawiki/extensions/Math] Add a standalone job, for T415230

2026-01-20

20:38 bd808: Cherry picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1229186 (T415113)
19:05 bd808: Rebooted deployment-cache-text08 to see if the mystery haproxy startup failure would go away (T415100)
18:50 bd808: Unblock 152.7.0.0/16 (T415100)

2026-01-17

23:32 ori: beta-scap with `php_l10n: true` completed successfully: https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/241466/console. PHP l10n files generated. Reverted local change to scap.cfg.
23:26 ori: Temporarily set `php_l10n: true` on deployment-deploy04:/etc/scap.cfg to see if next scap succeeds.

2026-01-16

16:33 dancy: Deleting deployment-mx03.deployment-prep (T412975)

2026-01-15

14:50 James_F: jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover-extensions/ArticleSummaries/ # T413232

2026-01-14

17:14 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226907
16:27 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1226893
15:57 bd808: Unblock 190.60.63.0/24 (T414541)

2026-01-13

15:04 James_F: Zuul: Make quibble-for-mediawiki-core-vendor-mysql-php84 voting, for T386108

2026-01-12

21:33 zabe: zabe@deployment-mwmaint03:~$ foreachwiki migrateLinksTable.php --table imagelinks # T413668
21:06 bd808: Unblock 66.81.168.0/21 (T414303)
17:42 dancy: Turned off instance deployment-prep.deployment-mx03
11:44 Lucas_WMDE: ssh integration-castor05.integration.eqiad1.wikimedia.cloud sudo -u jenkins-deploy rm -rf /srv/castor/castor-mw-ext-and-skins/master/mediawiki-node20 # fix failure seen in mediawiki-node20 46331, 46344

2026-01-10

21:48 taavi: reload zuul for https://gerrit.wikimedia.org/r/1224782
00:25 bd808: Unblock 91.160.0.0/12 (T414190)

2026-01-09

17:33 thcipriani: re-enabling beta update jobs after test bad extension-list T411516
17:09 thcipriani: disabling beta update jobs to test bad extension-list T411516)

2026-01-08

21:30 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224815 T414136
18:24 bd808: Unblock 89.80.0.0/12 (T414113)
15:55 dancy: Upgrading gitlab-runner to v18.5.0 on gitlab-cloud-runners. (T414053)

2026-01-07

23:17 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1082574 https://gerrit.wikimedia.org/r/1224157 https://gerrit.wikimedia.org/r/1224159
23:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/896311 T27482
23:06 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1224218
17:34 James_F: Zuul: Add new extensions: IssueTrackerLinks, PreviewLinks, and WikiRAG
17:34 James_F: Zuul: [labs/tools/heritage] Point to the task to drop 8.1 testing
15:09 James_F: Zuul: [labs/tools/heritage] Add testing in PHP 8.2+, not just PHP 8.1
15:03 James_F: Zuul: Even for extension-broken, don't offer PHP 8.1 testing
15:02 James_F: Zuul: Move quibble experimental sqlite/postgres tests to PHP 8.3

2026-01-06

16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223690 T411814
16:16 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/1223189 T411814
00:30 bd808: Unblock 85.134.128.0/17 (T413755)
00:02 bd808: Unblock 89.166.128.0/17 (T413702)

2026-01-05

23:57 bd808: Unblock 185.233.104.0/22 (T413472)
23:51 bd808: Unblock 45.62.112.0/21 (T413079)
23:44 bd808: Unblock 85.134.200.0/21 (T413067)
19:03 dancy: Updated buildkitd to v0.26.3 in gitlab-cloud-runners
14:27 taavi: reload zuul for 1223191
13:57 James_F: Zuul: [mediawiki/php/wmerrors] Enable PHP 8.5 testing, for T410921

Server Admin Log logged from #wikimedia-releng ^connect for Beta Cluster, Continuous integration and various other Release Engineering projects.