Release Engineering/SAL/Archive 4

2018-12-31

• 16:45 Hauskatze: gerrit/github: created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/CiteDrawer, w/ github and diffusion mirror; cf. mw.org request
• 14:33 Hauskatze: gerrit: created https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/MissedPages & associated extension-MissedPages group; as requested on mw.org
• 14:05 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-MissedPages to mirror similar new Gerrit repo

2018-12-22

• 13:30 Hauskatze: github: deleting https://github.com/wikimedia/labs-tools-signpost mirror | T212559

2018-12-21

• 11:17 hashar: new doc job is https://integration.wikimedia.org/ci/job/publish-to-doc1001/ and runs just after https://integration.wikimedia.org/ci/job/publish-on-contint1001/
• 11:17 hashar: updating all publish jobs to also publish to doc1001.eqiad.wmnet | https://gerrit.wikimedia.org/r/#/c/integration/config/+/480657/ | T137890

2018-12-20

• 18:53 thcipriani: unstick post-merge pipeline (beta-scap-eqiad stuck for 9 hours), [x] beta-scap-eqiad job clogging the works
• 05:10 legoktm: deployed https://gerrit.wikimedia.org/r/480908
• 05:04 legoktm: deployed https://gerrit.wikimedia.org/r/480904

2018-12-19

• 23:27 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
• 18:05 Hauskatze: Created mediawiki/extensions/ExternalGuidance.git repos on Gerrit, set-up mirrors on Diffusion and GitHub; per mediawiki.org request.
• 15:20 hashar: Testing spicerack postmerge job for T205894: contint1001$ zuul enqueue --trigger gerrit --pipeline postmerge --project operations/software/spicerack --change 480724,2

2018-12-18

• 10:56 legoktm: rebuilding docker tox-* images
• 08:17 legoktm: rebuilding docker tox image https://gerrit.wikimedia.org/r/480428

2018-12-17

• 22:50 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/480256
• 22:26 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/480165/
• 11:43 Amir1: ores:18d3657 is going beta

2018-12-14

• 16:19 Krenair: T204500 Powering off the remaining trusty and main deployment instance, deployment-zotero01, which is no longer in use anyway
• 13:08 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf | tee WikibaseQualityConstraints-config.php # T209957
• 06:23 kart_: Beta: Updated cxserver to de618f3

2018-12-13

• 21:56 thcipriani: integration-slave-jessie-1002:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
• 18:31 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/479501/
• 17:20 Amir1: ores:1a3de73 is going beta
• 09:32 hashar: Adding integration-slave-jessie-1001 back online
• 09:30 hashar: cleaned up disk on integration-slave-jessie-1001
• 01:05 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/479158

2018-12-12

• 21:50 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-ConditionalShowSection | refs. T211821
• 21:20 bearND: (beta): Update mobileapps to 55981a8
• 19:45 hashar: contint1001: sudo chown -R zuul:zuul /etc/zuul/wikimedia/.git
• 19:37 greg-g: enabled 2fa on the wmfgerrit github user account, recovery codes in releng's pwstore
• 17:24 Lucas_WMDE: lucaswerkmeister-wmde@deployment-mwmaint01:~$ mwscript extensions/WikibaseQualityConstraints/maintenance/ImportConstraintEntities.php --wiki=wikidatawiki --config-format=wgConf | tee WikibaseQualityConstraints-config.php # T209957

2018-12-11

• 20:03 hashar: building image docker-registry.wikimedia.org/releng/commit-message-validator:0.6.0 | T210278
• 10:09 hashar: jenkins: removed some legacy/unused user token via https://integration.wikimedia.org/ci/administrativeMonitor/legacyApiToken/manage | T206234
• 10:03 hashar: Deleting unused instance jenkinstest.integration.eqiad.wmflabs | T208803
• 09:42 godog: enable profile::rsyslog::udp_localhost_compat
• 09:04 hashar: gerrit: added _joe_ to 'integration' project so he can +2 changes notably to integration/config ( https://gerrit.wikimedia.org/r/#/admin/groups/10,members )

2018-12-09

• 23:38 Krenair: restarted apache on deployment-mediawiki-07 for T211524

2018-12-05

• 14:58 thcipriani: bring integration-slave-jessie-1004 back online (disks look fine)
• 02:57 kart_: Beta: Updated cxserver to c4240e6

2018-12-04

• 06:04 legoktm: deploying https://gerrit.wikimedia.org/r/477115
• 04:08 legoktm: deploying https://gerrit.wikimedia.org/r/473724 https://gerrit.wikimedia.org/r/477416

2018-12-03

• 22:05 Krenair: T210214 deleting deployment-cache-text04
• 15:01 Amir1: ores:03b9c98 is going beta

2018-11-30

• 22:32 thcipriani: removing deployment-redis0{5,6} since I can't ssh in, and they don't seem to have any puppet roles applied

2018-11-29

• 14:26 godog: switch mediawiki logging to use localhost syslog -> kafka -> logstash
• 13:54 hashar: Building docker-registry.discovery.wmnet/releng/java8-sonar-scanner:0.1.0 for https://gerrit.wikimedia.org/r/475496 | T209849
• 08:35 hashar: Deleted instances castor02 and integration-publishing , replaced by new instances in the new WMCS region | T208803

2018-11-28

• 08:45 hashar: Switching Jenkins job cache to integration-castor03 with AN EMPTY CACHE | T208803
• 08:18 godog: test mediawiki kafka logging on deployment-mediawiki-07

2018-11-27

• 15:55 Amir1: giving User:Fisch-WMDE and User:Thiemo Mättig (WMDE) admin rights on enwiki, dewiki and arwiki (WMDE staff, beta cluster)
• 13:59 godog: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/475352
• 11:38 Amir1: Deploying ores:9b9ba06 to beta

2018-11-26

• 22:22 Krenair: shutoff deployment-cache-text04, now replaced with deployment-cache-text05 in the new region - T210214
• 18:02 Amir1: deploy ores:3cdaaa6 to beta

2018-11-23

• 13:55 dcausse: restarted elasticsearch on all deployement-elastic0X nodes (search broken on the beta cluster)
• 11:09 hashar: Jenkins: removing plugins "Single Use Slave" and "Event Publisher (via ZMQ PUB SUB)". Were used for Nodepool | T209361
• 10:29 hashar: Building releng/java8-wikidata-query-rdf:0.2.1 container for https://gerrit.wikimedia.org/r/#/c/integration/config/+/475428/ T209776

2018-11-22

• 10:34 hashar: Building container releng/java8-wikidata-query-rdf:0.2.0 to support Chrome ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/475298/ | T209776 )

2018-11-21

• 21:44 Krenair: Changed beta.wmflabs.org and *.beta.wmflabs.org A records from 208.80.155.135 (deployment-cache-text04) to 185.15.56.36 (deployment-cache-text05)
• 20:49 hashar: gerrit: added WMDE-leszek to group "Gerrit Managers" https://gerrit.wikimedia.org/r/#/admin/groups/119,members | T200311
• 03:25 bd808: Forced puppet run on deployment-mediawiki-0[79] to pick up new redis::shards settings T210030
• 03:02 bd808: Repeated application of role::mediawiki::memcached on deployment-memc04, deployment-memc06, and deployment-memc07 for T210030
• 01:22 bd808: Applied role::mediawiki::memcached on deployment-memc05.deployment-prep.eqiad.wmflabs to provision redis T210030

2018-11-20

• 23:45 mutante: deployment-deploy01 edited /srv/deployment/iegreview/iegreview/.git/DEPLOY_HEAD - - replaced deployment-tin with deployment-deploy1 to fix scap cloning / puppet
• 23:01 mutante: deployment-deploy01 edited /srv/deployment/scholarships/scholarships/.git - replaced deployment-tin with deployment-deploy1 to fix scap / cloning of scholarships app
• 22:42 thcipriani: repooling integration-slave-jessie-1003 after cleaning mvn and gradle cache
• 20:23 Krenair: Changed deployment-prep mysql repl password in attempt to get replication working again, have stored it at deployment-puppetmaster03:/var/lib/git/labs/private/modules/secret/secrets/mysql/repl_password
• 18:18 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@7553087]
• 14:59 hashar: created integration-castor03.integration.eqiad.wmflabs intended as a replacement for castor02 | T208803
• 14:34 hashar: beta-scap job fixed by deleting a file on deployment-mediawiki-07
• 14:27 hashar: deployment-mediawiki-07 : sudo rm -fR /srv/mediawiki/.~tmp~/
• 13:55 andrewbogott: deleting deployment-redis05 and deployment-redis06 as per Giuseppe, "we're not using the old jobqueue, we should remove those vms"
• 13:08 twentyafterfour: removed local unix user mwdeploy from deployment-mediawiki-07 because it was shadowing the real mwdeploy user in ldap
• 13:01 twentyafterfour: PHP Startup: Unable to load dynamic library '/usr/lib/php/20151012/luasandbox.so' - /usr/lib/php/20151012/luasandbox.so: cannot open shared object file: No such file or directory T208101
• 12:11 twentyafterfour: scap failures on deployment-mediawiki-07 are related to uid/gid mismatch of the mwdeploy user, specifically the owner of that user's home dir is uid 603 but /etc/passwd|group have a different uid/gid for the same username. T208101
• 11:47 hashar: Armed keyholder on deployment-deploy01 Got shutdown while being migrated a new cloud region # T208101
• 11:00 hashar: deployment-deploy01 got migrated to a new region but the Jenkins configuration had not been updated. Adjusting IP address from 10.68.23.38 to 172.16.4.18 | T208101
• 10:58 hashar: Clearing out deployment-deploy01 disk space. Went offline due to disk space consumption
• 10:04 Krenair: manually fixed deployment-mediawiki-09:/srv/mediawiki/wmf-config/db-labs.php to match deployment copy, not sure why it didn't deploy properly yet

2018-11-19

• 22:40 Krenair: manually sorted certs for deployment-puppetdb02
• 22:23 Krenair: duplicated wiki(user|admin) mysql users on deployment-db0[34] - previous hosts 10.%, new hosts 172.16.%
• 18:32 Krenair: creating deployment-cache-text05 to replace deployment-cache-text04
• 15:53 Amir1: cherry-picking gerrit:474694/1 in beta puppetmaster
• 15:06 Amir1: ores:e957b24 is going beta
• 13:47 hashar: Shutdown integration-publishing , replaced by integration-publishing02 # T208803
• 13:21 hashar: Created integration-publishing02 172.16.4.5 for WMCS region migration # T208803
• 11:13 hashar: updating jobs wikidata-query-rdf-maven-java8-docker wikidata-query-rdf-maven-java8-docker-site-publish for https://gerrit.wikimedia.org/r/#/c/integration/config/+/474660/
• 10:23 gehel: Updating docker-pkg files on contint1001 for wikdiata-query-rdf image

2018-11-17

• 06:26 legoktm: deployed https://gerrit.wikimedia.org/r/474415

2018-11-16

• 21:01 legoktm: deploying https://gerrit.wikimedia.org/r/473718
• 20:51 legoktm: deploying https://gerrit.wikimedia.org/r/473671 https://gerrit.wikimedia.org/r/474330
• 20:36 legoktm: deployed https://gerrit.wikimedia.org/r/474306
• 17:37 hashar: Added integration-slave-docker-1046 and integration-slave-docker-1047
• 17:20 hashar: Deleted integration-slave-docker-1045 (wrong OS stretch, must be jessie)
• 17:19 hashar: Added two more Jenkins docker slaves: https://integration.wikimedia.org/ci/computer/integration-slave-docker-1044/ and https://integration.wikimedia.org/ci/computer/integration-slave-docker-1045/
• 14:35 hashar: Building docker image releng/java8:0.4.2
• 13:33 hashar: building docker image releng/java8:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/472661/

2018-11-15

• 21:09 hashar: Last Nodepool instance had id 1099516 (yeah more than a million)
• 17:15 Amir1: ores:51cdf6b is going beta

2018-11-14

• 18:48 andrewbogott: moving deployment-mediawiki-07 to labvirt1008
• 18:31 andrewbogott: moving deployment-chromium01 to labvirt1009
• 18:06 andrewbogott: moving deployment-mx02 to labvirt1003
• 18:05 andrewbogott: migrating deployment-snapshot01 to labvirt1001
• 10:39 gtirloni: added gtirloni as projectadmin in deployment-prep project

2018-11-13

• 22:39 thcipriani: reenabling and running puppet on deployment-deploy01
• 22:19 andrewbogott: moving deployment-urldownloader02 to labvirt1012
• 22:02 thcipriani: disable puppet on deployment-deploy01 temporarily while deployment-deploy02 is migrating to preserve dsh files
• 21:59 andrewbogott: moving deployment-deploy02 to another labvirt
• 21:55 andrewbogott: moving deployment-webperf12 to a new labvirt
• 21:50 andrewbogott: moving deployment-dumps-puppetmaster02 to a new labvirt
• 21:43 andrewbogott: moving deployment-elastic05 to a new labvirt to clear out labvirt1016
• 13:01 arturo: a puppet refactor for the aptly module may have caused some puppet issues. Should be solved now
• 09:31 addshore: manually brought integration-slave-docker-1021 back online
• 09:29 addshore: integration-slave-docker-1021:/# docker rmi $(docker images | grep " months " |grep -v " [1-2] months " | awk '{print $3}')
• 09:26 addshore: integration-slave-docker-1021:/# docker rmi $(docker images | grep " months " |grep -v " [1-5] months " | awk '{print $3}')

2018-11-12

• 12:30 hashar: mass purging legacy Jenkins jobs from contint1001. They were still deployed but no more in JJB, that causes the logs / artifacts to be left behind. | T209123
• 11:26 hashar: building container releng/operations-puppet:0.5.1 to bump mtail version ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/472960/ by godog )

2018-11-11

• 23:47 addshore: manually set quibble-vendor-mysql-hhvm-docker timeout to be 60 (i should really get my jjb stuff working again)

2018-11-09

• 13:43 Amir1: ores:0728805 is going beta
• 12:13 kart_: Update cxserver to 01686f6

2018-11-08

• 22:55 thcipriani: contint1001: cleanup with (essentially, but not exactly) https://phabricator.wikimedia.org/P7781
• 22:21 thcipriani: contint1001:sudo find /srv/jenkins/builds -maxdepth 2 -mindepth 2 -type d -regex '.*/[1-9][0-9]+' -mtime +500 -exec rm -r "{}" \;
• 21:50 thcipriani: contint1001:rm apps-android-wikipedia-test builds > 30 days old
• 15:42 Krenair: dropped https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/434055/ cherry-pick and rebased to fix puppet updates
• 15:37 addshore: for https://gerrit.wikimedia.org/r/472476
• 15:36 addshore: addshore@deployment-deploy01:~$ sudo puppet agent -tv
• 14:30 hashar: updated search-highlighter-maven-java8-docker-site-publish to decouple install and site goals | T208496

2018-11-07

• 23:00 thcipriani: repooling integration-slave-docker-1017 after cleaning up docker images
• 19:09 Amir1: ores:25dfa4f is going to beta cluster
• 09:06 hashar: building container releng/operations-puppet:0.5.0 for python3 | T208873

2018-11-06

• 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/472044
• 15:25 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/blubber/+/471277/

2018-11-04

• 17:53 legoktm: deploying https://gerrit.wikimedia.org/r/471251
• 17:50 legoktm: deployed https://gerrit.wikimedia.org/r/471283

2018-11-02

• 19:54 Amir1: stopping redis-server on deployment-ores01, this would make ores go down in beta cluster. Testing for T181632
• 14:04 Krenair: made onimisionipe a projectadmin per request in -cloud
• 02:30 legoktm: deploying https://gerrit.wikimedia.org/r/471198
• 00:38 legoktm: deploying https://gerrit.wikimedia.org/r/471186 https://gerrit.wikimedia.org/r/471181

2018-11-01

• 17:12 legoktm: deploying https://gerrit.wikimedia.org/r/471047
• 15:55 legoktm: deploying https://gerrit.wikimedia.org/r/471033 https://gerrit.wikimedia.org/r/471034
• 15:41 legoktm: deploying https://gerrit.wikimedia.org/r/471032
• 15:29 legoktm: deploying https://gerrit.wikimedia.org/r/471029
• 15:11 legoktm: building docker for https://gerrit.wikimedia.org/r/471025
• 14:48 andrewbogott: moving deployment-redis05 to labvirt1012
• 14:47 Krenair: shut off deployment-redis05 for migration to new physical host
• 14:45 legoktm: building mediawiki-tarball docker image https://gerrit.wikimedia.org/r/471019

2018-10-31

• 21:16 Krenair: remove horizon hiera config for deployment-redis0[56] to unbreak puppet and remove old redis0[12] instance IPs T208040
• 19:55 andrewbogott: moving deployment-elastic06 to labvirt1012
• 19:40 andrewbogott: moving deployment-cpjobqueue to labvirt1012 to help clear out labvirt1017
• 19:11 andrewbogott: moving deployment-kafka-jumbo-1 to labvirt1012 to help clear out labvirt1017
• 18:54 andrewbogott: moving deployment-kafka-main-2 to labvirt1012 to help clear out labvirt1017
• 17:21 Amir1: ores:70ba14b is going to beta
• 13:23 godog: enable statsd reporting for swift

2018-10-30

• 16:51 hashar: building docker container releng/operations-puppet:0.4.1 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/470574/
• 14:48 addshore: beta T208307: wikiadmin@10.68.23.30(wikidatawiki)> delete from wb_changes_dispatch where chd_site = 'specieswiki'
• 11:15 hashar: contint1001: cleaned up old Docker images
• 10:13 hashar: Building Docker container releng/operations-puppet:0.4.0 for https://gerrit.wikimedia.org/r/470419 by _joe_

2018-10-29

• 22:56 legoktm: deployed https://gerrit.wikimedia.org/r/470453 https://gerrit.wikimedia.org/r/469954 https://gerrit.wikimedia.org/r/470241 https://gerrit.wikimedia.org/r/468900 https://gerrit.wikimedia.org/r/469181
• 22:54 James_F: Manually triggered a beta-update-databases-eqiad run to see if our fix for T208043 has worked.
• 10:04 hashar: cleaned disk on integration-slave-jessie-1002
• 04:07 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
• 00:44 legoktm: also deploying https://gerrit.wikimedia.org/r/470059
• 00:44 legoktm: deploying https://gerrit.wikimedia.org/r/469948

2018-10-26

• 09:04 hashar: jenkins: update IRC config at https://integration.wikimedia.org/ci/configure to replace #wikimedia-ri-ci by #wikimedia-infrastructure ( https://gerrit.wikimedia.org/r/#/c/integration/config/+/469034/ )

2018-10-25

• 17:21 bearND: (beta): Update mobileapps to 58cbdff
• 07:50 hashar: enabling puppet again on deployment-deploy01 . Was disabled by _joe_ for apache-fast-test hacking

2018-10-24

• 20:44 hasharDinner: Rebuilding CI containers for Quibble 0.0.28
• 20:27 hasharDinner: tagged Quibble 0.0.28 at 1ac8fe3
• 08:57 hashar: gerrit: added Lars "liw" Wirzenius to the Administrators group | T207830
• 07:27 Krenair: T207825 reapplied role::jobqueue_redis::master to deployment-redis prefix
• 07:01 Krenair: T207825 replacing deployment-redis3-changeprop with deployment-redis3-changeprop02 (jessie m1.small)
• 06:59 Krenair: T207825 moved role::jobqueue_redis::master role from deployment-redis prefix to deployment-redis0[56]
• 05:22 kart_: Beta: Updated cxserver to 9ad60d9

2018-10-23

• 23:59 legoktm: deploying https://gerrit.wikimedia.org/r/468171

2018-10-22

• 13:01 kart_: Beta: Update cxserver to 7f996f3
• 09:17 _joe_: deployment-prep: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/468583/, disable puppet on deployment-mediawiki-09
• 00:27 Krenair: Added gtirloni as a member per T207474 - I imagine he'll want to get in to look at shinken-related things

2018-10-21

• 21:27 Krenair: rm'd scap l10n stuff on deployment-deploy01 as usual
• 12:25 legoktm: deployed https://gerrit.wikimedia.org/r/468816

2018-10-19

• 11:36 hashar: nuking some caches on castor02.integration.eqiad.wmflabs to free up disk space
• 07:08 legoktm: also deploying https://gerrit.wikimedia.org/r/c/integration/config/+/468022
• 07:07 legoktm: deploying https://gerrit.wikimedia.org/r/467329 https://gerrit.wikimedia.org/r/468417 https://gerrit.wikimedia.org/r/468393 https://gerrit.wikimedia.org/r/467724

2018-10-18

• 13:27 Amir1: deploying ores:d724d20

2018-10-16

• 19:09 twentyafterfour: Branching 1.32.0-wmf.26
• 19:08 legoktm: integration-slave-docker-1017 back online
• 19:07 legoktm: root@integration-slave-docker-1017:/# docker rmi $(docker images -q) to clean up disk space
• 04:22 legoktm: deploying https://gerrit.wikimedia.org/r/467563
• 04:14 legoktm: deploying https://gerrit.wikimedia.org/r/467562
• 04:04 legoktm: deploying https://gerrit.wikimedia.org/r/467560
• 03:06 Krinkle: Rebooted deployment-deploy01 Jenkins slave (stuck for 8 hours)

2018-10-15

• 22:28 legoktm: deployed https://gerrit.wikimedia.org/r/467536
• 22:20 legoktm: fixed disk space issue with integration-slave-jessie-1004 and brought it back online
• 22:16 legoktm: legoktm@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf *
• 22:09 legoktm: triggering jobs on contint1001 via gear_client, kill if necessary
• 22:03 legoktm: building Quibble PHP 7.3 images (https://gerrit.wikimedia.org/r/467259)
• 20:31 bearND: Beta cluster: Update mobileapps to c2a4ef9
• 19:53 legoktm: rebuilding npm* docker images for https://gerrit.wikimedia.org/r/467451
• 18:13 legoktm: deploying https://gerrit.wikimedia.org/r/467434
• 17:56 legoktm: building php73(-compile)? images https://gerrit.wikimedia.org/r/465204
• 15:14 thcipriani: bring integration-slave-jessie-1001 back online
• 13:30 hashar: Cleaned up disk space on integration-slave-jessie-1001
• 05:10 kart_: Update cxserver to b51f363
• 04:00 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n*

2018-10-13

• 04:45 legoktm: rebuilding docker images for https://gerrit.wikimedia.org/r/467023
• 04:34 legoktm: deploying https://gerrit.wikimedia.org/r/467022
• 03:20 Krinkle: Fixed console section pattern for "setup quibble mw-update"
• 03:19 Krinkle: Fixed console section pattern for "setup quibble npm-ls"
• 03:04 legoktm: deploying https://gerrit.wikimedia.org/r/467016
• 02:15 legoktm: deploying https://gerrit.wikimedia.org/r/467012
• 01:55 legoktm: building quibble-stretch-php72 image https://gerrit.wikimedia.org/r/466991
• 01:25 legoktm: rolling out https://gerrit.wikimedia.org/r/c/integration/config/+/466950

2018-10-12

• 21:13 legoktm: deployed https://gerrit.wikimedia.org/r/465049
• 20:32 Krinkle: deploy01 /usr/local/bin/mwscript update.php --wiki=simplewiki --quick
• 20:32 Krinkle: deploy01 /usr/local/bin/mwscript update.php --wiki=ruwiki --quick
• 20:14 mutante: contint1001 - gzip merge-debug.log.2018-* debug.log.2018-* etc.. all in /var/log/zuul/ there seems to be no compression (but max age is 1 month)
• 20:11 mutante: contint1001 - gzip zuul.log.2018-* in /var/log/ to prevent running out of disk
• 20:08 mutante: contint1001 - apt-get clean - frees 1GB to bring it _just_ under the Icinga warning threshold :p
• 19:23 legoktm: rebuilding docker images for https://gerrit.wikimedia.org/r/465606
• 17:18 marxarelli: bringing integration-slave-docker-1021 back online
• 17:16 marxarelli: killing 7 long-running containers (> 1 hour) on docker integration nodes (T198517)
• 16:42 marxarelli: killing 9 long-running containers (> 1 hour) on docker integration nodes (T198517)
• 16:31 marxarelli: taking integration-slave-docker-1021 offline (host is unresponsive)
• 16:25 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 321 affected jobs (T198517)
• 16:05 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465671 for 1 job mediawiki-quibble-vendor-mysql-php70-docker (T198517)
• 15:30 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 320 affected jobs
• 15:26 marxarelli: killing 51 long-running containers (> 1 hour) on docker integration nodes (T198517)
• 13:22 Reedy: running foreachwiki update.pph
• 11:48 Amir1: ores:ba4ea40 is going beta
• 05:14 legoktm: deployed https://gerrit.wikimedia.org/r/466821
• 04:40 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/466812
• 03:52 legoktm: deployed https://gerrit.wikimedia.org/r/466811
• 03:46 legoktm: tagged 0.0.27 of quibble (ec8d280)

2018-10-11

• 23:50 marxarelli: rolling back deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/465668 (full deployment delayed until around 1600 tomorrow)
• 23:45 marxarelli: killed 6 long-running containers (> 3 hours) on docker integration nodes (T198517)
• 23:34 marxarelli: redeploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job mediawiki-quibble-vendor-mysql-php70-docker
• 23:14 marxarelli: rolling back deployment of https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job
• 23:08 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/465668 for 1 job mediawiki-quibble-vendor-mysql-php70-docker
• 19:28 thcipriani: integration-slave-jessie-1003 back online
• 19:28 thcipriani: integration-slave-jessie-1003:sudo rm -rf /srv/jenkins-workspace/workspace/*
• 18:18 Amir1: ores:1a5630f is going beta
• 18:17 marxarelli: killed 42 long-running (> 3 hours) containers on docker integration nodes (T198517)
• 18:16 marxarelli: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/463852/ to all 320 affected jobs
• 17:47 marxarelli: killing long-running containers (> 3 hours) on docker integration nodes (T198517)
• 17:37 marxarelli: deploying https://gerrit.wikimedia.org/r/c/integration/config/+/463852/ to 8 wmf-quibble-* jobs
• 02:03 Krinkle: Fix Jenkins' section pattern for 'setup-docker-quibble-castor-load'
• 02:03 Krinkle: Fix Jenkins' collapsible sections for "npm-install" and "mw-install" under quibble to match current patterns

2018-10-10

• 19:38 awight: restarted ORES celery workers on ores2003 (~17:00), ores200* (17:05)
• 17:15 twentyafterfour: cowboy-coding on deployment-deploy01 to solve scap fatal error dcheck failing to catch fatals

2018-10-09

• 19:00 Krinkle: Re-enable beta-scap-eqiad job
• 18:19 Krinkle: Messing with scap in beta to test T121597 / D1114
• 15:32 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*

2018-10-08

• 03:31 kart_: Update(d) cxserver to 47a864b
• 03:02 legoktm: onlined integration-slave-jessie-1002
• 02:20 legoktm: legoktm@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2018-10-06

• 13:59 Reedy: cleared some large folders out of /tmp on deployment-deploy01

2018-10-05

• 19:47 marxarelli: bringing integration-slave-docker-1040 back online
• 19:05 marxarelli: taking integration-docker-slave-1040 offline for docker daemon restart
• 19:04 marxarelli: bringing integration-slave-docker-1038/1041/1043 back online
• 19:02 marxarelli: taking integration-docker-slave-1038/1041/1043 offline for docker daemon restart
• 19:01 marxarelli: bringing integration-slave-docker-1033/1037 back online
• 18:58 marxarelli: taking integration-docker-slave-1033/1037 offline for docker daemon restart
• 18:56 marxarelli: bringing integration-slave-docker-1034 back online
• 18:56 marxarelli: integration-puppetmaster01:/var/lib/git/operations/puppet is up-to-date again after manually updating submodules and subsequent automated git-sync-upstream
• 18:33 marxarelli: taking integration-slave-docker-1034 offline for docker daemon restart
• 11:18 Krenair: rm -rf /tmp/scap_l10n_* on deployment-deploy01
• 06:00 legoktm: deployed https://gerrit.wikimedia.org/r/464757
• 04:35 legoktm: deploying https://gerrit.wikimedia.org/r/464747
• 01:17 legoktm: triggering php71 quibble jobs manually via contint1001

2018-10-04

• 22:23 thcipriani: updated jenkins jobs: https://phabricator.wikimedia.org/P7634
• 22:19 thcipriani: updating 103 jenkins jobs that use ci-src-setup-simple
• 21:45 thcipriani: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/464719/
• 21:13 marxarelli: bringing integration-slave-jessie-1003 back online
• 21:12 marxarelli: deleted /mnt/home/jenkins-deploy/{.m2,.gradle} on integration-slave-jessie-1003
• 21:10 marxarelli: deleting cache directories in /mnt/home/jenkins-deploy on integration-slave-jessie-1003 to free up disk space
• 20:31 marxarelli: terminating long-running (> 3 hours) CI docker containers (T198517)

2018-10-03

• 18:45 marxarelli: deploying I0fcd95 for all 313 affected jobs
• 18:16 marxarelli: deploying I0fcd95 for 12 mediawiki-quibble-vendor-* jobs
• 17:46 marxarelli: removing old workspaces on integration-slave-docker-1034 (`rm -rf /srv/jenkins-workspace/workspace/*`) and bringing back online
• 16:52 thcipriani: reloading zuul to deploy Change gate-and-submit-l10n to low precedence
• 15:46 marxarelli: bringing integration-slave-docker-1040/1043 nodes back online after killing long running docker job and freeing up /var/lib/docker space (T206134)
• 15:40 marxarelli: killing long running docker jobs on integration-slave-docker-1040/1043 which are filling up /var/lib/docker with log output
• 15:12 thcipriani: reloading zuul to deploy l10 pipeline
• 14:53 thcipriani: integration-slave-docker-1038 removed workspaces, brought back online integration-slave-docker-104{0,3} need more investigation since docker is full(?)
• 07:07 mdholloway: deployment-maps04 deployed [kartotherian/deploy@27062b4]: Specify WDQS endpoint in the service config (T205607)
• 06:22 _joe_: cherry-picking 455154 (php-fpm installation) on deployment-prep

2018-10-02

• 21:34 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/464048/
• 18:22 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_*
• 18:15 legoktm: deploying https://gerrit.wikimedia.org/r/464000
• 16:45 thcipriani: bring integration-slave-docker-1037 back online after rm -rf /srv/jenkins-workspace/workspace/*
• 16:20 thcipriani: investigating integration-slave-docker-1037
• 08:56 godog: bounce logstash
• 07:59 legoktm: deployed https://gerrit.wikimedia.org/r/463905
• 07:38 legoktm: building quibble-stretch-php71 docker image
• 05:32 legoktm: rebuilding quibble-stretch images https://gerrit.wikimedia.org/r/463883
• 03:26 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463886

2018-10-01

• 20:09 thcipriani: deployment-deploy01:sudo rm -rf /tmp/scap_l10n_* to remove stale l10n json and free up space
• 17:13 marxarelli: bringing integration-slave-docker-1041 back online following source directory clean up (T205902)
• 16:52 marxarelli: removing old workspace src directories left by non-quibble docker jobs on integration-slave-docker-1041
• 07:10 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest
• 05:24 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@07cbfb4]: Update mobileapps to a1fa41b

2018-09-29

• 14:48 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/463615
• 01:22 Krinkle: Marking integration-slave-docker-1041 as offline (again). Why/How did it come back?
• 00:24 Krinkle: Marking integration-slave-docker-1041 as offline. Various odd build failures, including https://integration.wikimedia.org/ci/job/mediawiki-quibble-composer-mysql-php70-docker/6913/console

2018-09-28

• 15:31 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/CentralAuth/maintenance/deleteLocalPasswords.php --wiki=fawiki --prefix (T201009)
• 14:50 thcipriani: investigating integration-slave-docker-1041
• 07:35 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@bf09080]: Update mobileapps to 7878ffc

2018-09-27

• 17:53 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@a0054ba]: Update mobileapps to 0d6c2b7
• 07:38 mdholloway: deployment-maps04 updated tilerator and kartotherian node modules (T195513, T200594)

2018-09-26

• 15:13 thcipriani: integration-slave-docker-1034:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online -- https://phabricator.wikimedia.org/P7592
• 15:05 thcipriani: integration-slave-docker-1033:sudo rm -rf /srv/jenkins-workspace/workspace/* and bring back online
• 14:47 thcipriani: investigating integration-slave-docker-103{3,4}
• 11:57 Amir1: gerrit:462927 (ores) is going to beta
• 08:24 hashar: Restarting CI Jenkins on contint1001 [#2]
• 08:14 hashar: Restarting CI Jenkins on contint1001

2018-09-25

• 23:01 marxarelli: configured new jenkins node integration-slave-docker-1043 with 6 executors
• 23:01 marxarelli: replaced integration-slave-docker-1042 with new integration-slave-docker-1043 instance
• 22:39 marxarelli: launching new integration-slave-docker-1042 bigram instance
• 22:33 marxarelli: deleting remaining m1.medium instances used as m4executors (T205362)
• 22:15 marxarelli: taking remaining m1.medium m4executor jenkins nodes offline (T205362)
• 18:16 marxarelli: reconfiguring bigram jenkins nodes to use 6 executors. 7 were configured by mistake (T205362)
• 18:00 marxarelli: configuring new integration-slave-docker-1041 jenkins node with 7 executors (T205362)
• 17:42 marxarelli: configuring new jenkins node integration-slave-docker-1040 with 7 executors (T205362)
• 17:38 marxarelli: launching integration-slave-docker-1041 bigram instance (T205362)
• 17:30 marxarelli: the puppet parameter for docker_lvm_volume specified in horizon was not applied correctly on the first puppet run for some reason. tearing down integration-slave-docker-1039...
• 17:25 marxarelli: launching integration-slave-docker-1040 bigram instance (T205362)
• 17:24 marxarelli: deleting instances integration-slave-docker-1007/1008 (T205362)
• 17:13 marxarelli: launching new integration-slave-docker-1039 bigram instance
• 17:12 marxarelli: taking integration-slave-docker-1007/1008 offline for replacement (T205362)
• 17:09 marxarelli: deleting integration-slave-docker-1030/1031 instances (T205362)
• 17:05 marxarelli: taking integration-slave-docker-1030/1031 offline for replacement
• 16:47 marxarelli: increasing executors to 7 for jenkins nodes integration-slave-docker-1033/1034
• 16:46 marxarelli: new instance creation delayed due to quota
• 16:45 marxarelli: launching new integration-slave-docker-1039/1040 bigram instances
• 01:21 legoktm: deployed https://gerrit.wikimedia.org/r/450508
• 00:36 legoktm: deploying https://gerrit.wikimedia.org/r/462609
• 00:22 legoktm: deploying https://gerrit.wikimedia.org/r/453447

2018-09-24

• 20:21 bearND: (beta): Update mobileapps to badb463
• 10:55 hashar: gerrit: granting labs/tools/* project owners the ability to submit changes | https://gerrit.wikimedia.org/r/#/c/labs/tools/+/462420/
• 09:51 hashar: deployment-deploy01 : backed up /srv/mediawiki-staging/php-master/cache/gitinfo and created a new. Its size of 69632 bytes might cause slow writes?? | T204762
• 09:24 hashar: Live hacked scap code on deployment-deploy01 for T204762 and reverted hack changes
• 08:32 hashar: deployment-deploy01 rm -fR /tmp/scap_l10n_*
• 06:41 legoktm: deploying https://gerrit.wikimedia.org/r/462341
• 03:45 kart_: Update cxserver to d913793

2018-09-23

• 14:03 Krenair: rm stuff in deployment-deploy01:/tmp to try to clear space and stop shinken whining
• 01:05 andrewbogott: rebooted deployment-maps03; OOM and also T205195

2018-09-22

• 20:51 Hauskatze: github: deleting several wikimedia/mediawiki-extensions-Collection-.* mirror repos for T183891
• 20:05 Hauskatze: github: deleted mirror wikimedia/mediawiki-extensions-Collection-OfflineContentGenerator-zim_renderer | T183891; moving to the next one
• 18:21 Krenair: went to do the same with deployment-maps03 and accidentally broke SSH access to the server
• 18:21 Krenair: removed ferm package from deployment-snapshot01 as it appeared unmanaged by puppet and was causing problems with SSH access from the current deployment hosts (previous logs referenced T153468, this just explains why puppet hadn't purged stuff)
• 18:01 Krenair: rm deployment-maps03:/etc/ferm/conf.d/10_redis_exporter_6379 as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)
• 18:00 Krenair: rm deployment-snapshot01:/etc/ferm/conf.d/10_prometheus-nutcracker-exporter as it was breaking ferm from starting (T153468), puppet has not re-created it so I assume it was historical (shouldn't puppet be purging such files?)

2018-09-21

• 17:26 marxarelli: adding jenkins node integration-slave-docker-1038 with 7 executors
• 16:47 marxarelli: added new jenkins node integration-slave-docker-1037 with 7 executors
• 15:49 marxarelli: replacing integration-slave-docker-1036 with new bigram instance
• 15:48 marxarelli: taking node integration-slave-docker-1035 offline due to unusually high steal cpu time and long build durations
• 15:17 marxarelli: integration-slave-docker-1035/1036 showing unusually high cpu steal and unusually long mean build durations
• 15:15 marxarelli: taking integration-slave-docker-1036 offline due to unusually high cpu steal % trend
• 15:13 marxarelli: launching integration-slave-docker-1037 bigram instance
• 13:03 Amir1: ores:7b987a7 is going beta
• 05:32 legoktm: deployed https://gerrit.wikimedia.org/r/461510

2018-09-20

• 23:48 marxarelli: adding new integration-slave-docker-1035/1036 jenkins nodes, each with 7 executors
• 23:23 marxarelli: launching integration-slave-docker-1035/1036 bigram instances
• 23:20 marxarelli: taking integration-slave-docker-1004/1005 offline for replacement (T202160)
• 16:52 Amir1: deploy ores:ee2d28b
• 11:21 hashar: Refreshing jenkins jobs to get rid of docker run option "--tmp /tmpfs" . It is mounted with 'noexec' which causes various jobs to fail. | T203181 and T204919
• 11:17 hashar: deployment-deploy01: removed /srv/deployment/analytics/refinery-cache (8GBytes)
• 11:07 hashar: deployment-deploy01 is out of disk space (again)

2018-09-19

• 21:18 Hauskatze: github: deleted https://github.com/wikimedia/mediawiki-services-ocg-collection | T183891
• 20:07 bearND: (beta): Update mobileapps to a224e99
• 18:56 Amir1: ores:76fe25a goes to beta (T204862)

2018-09-18

• 10:04 hashar: Updating Quibble Jenkins jobs to 0.0.26
• 09:46 hashar: updating mwselenium-quibble-docker to Quibble 0.0.26
• 07:48 hashar: Updating jenkins jobs to use Quibble 0.0.25
• 07:34 hashar: deployment-sca01: rm -fR /srv/ores /srv/deployment/cxserver.jenkins # untouched since 2016
• 07:31 hashar: cleaning disk on deployment-sca01
• 00:30 marxarelli: configuring integration-slave-docker-1034 jenkins node to use 6 executors

2018-09-17

• 22:06 Hauskatze: added missing log entries (actor, etc.) on AbuseFilter (addMissingLoggingEntries.php) for beta
• 22:00 Reedy: update.php in screen is done
• 21:53 Hauskatze: maurelio@deployment-deploy01:~$ mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force (998 nodes loaded)
• 21:12 Reedy: running `foreachwiki update.php --quick` in screen on deployment-deploy01
• 20:17 bearND: (beta): Update mobileapps to d56e4cf
• 20:14 marxarelli: running `jenkins-jobs update config/ 'service-pipeline*'` to deploy I5df2b7
• 19:37 hasharAway: jenkins: remove compiler02.puppet3-diffs.eqiad.wmflabs and compiler03.puppet3-diffs.eqiad.wmflabs from jenkins config. Instances deletd
• 17:25 mdholloway: deployment-maps04 updated kartotherian and tilerator to latest (T109776)
• 13:24 hashar: Building docker images for Quibble 0.0.25 | https://gerrit.wikimedia.org/r/460536
• 12:58 hashar: mwselenium-quibble-docker to Chromium 69 | https://gerrit.wikimedia.org/r/#/c/integration/config/+/460512/ | T204214
• 12:27 phuedx: phuedx@deployment-deploy01:~$ foreachwikiindblist gettingstarted-with-category-suggestions.dblist extensions/GettingStarted/maintenance/populate_categories.php
• 09:15 Amir1: ladsgroup@deployment-deploy01:~$ foreachwikiindblist all-labs populateChangeTagDef.php --set-user-tags-only --force
• 07:04 legoktm: building npm6 docker images: https://gerrit.wikimedia.org/r/453441 https://gerrit.wikimedia.org/r/451690 https://gerrit.wikimedia.org/r/453445
• 00:10 legoktm: deployed https://gerrit.wikimedia.org/r/460775

2018-09-16

• 23:01 legoktm: deployed https://gerrit.wikimedia.org/r/460774 https://gerrit.wikimedia.org/r/#/c/integration/config/+/460578/
• 16:26 Hauskatze: phabricator: Account https://phabricator.wikimedia.org/p/Ponani/ disabled by me due to vandalism | T200856
• 12:47 Hauskatze: github: deleting <https://github.com/wikimedia/mediawiki-extensions-CommunityTwitter> | T204372

2018-09-14

• 17:47 marxarelli: adding instance-type-* labels to m4executor nodes in jenkins
• 14:27 hashar: deployment-deploy01 /srv/mediawiki synced
• 14:16 hashar: deployment-deploy01: sudo rm -fR /srv/mediawiki && mkdir /srv/mediawiki && chown mwdeploy:mwdeploy /srv/mediawiki && scap pull
• 13:57 hashar: deployment-deploy01: mv /srv/mediawiki/php-master/cache/l10n/*.php /srv/mediawiki/php-master/cache/l10n/backup-php/
• 13:55 hashar: deployment-deploy01 13:53:58 rsync: write failed on "/srv/mediawiki/php-master/cache/l10n/upstream/l10n_cache-ab.cdb.json": No space left on device (28)
• 13:48 hashar: Tagged Quibble 0.0.25 01663f5
• 11:49 hashar: Cleaned /tmp on deployment-deploy01
• 11:31 hashar: Manually running https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/28313/ | T204340
• 05:50 legoktm: deployed https://gerrit.wikimedia.org/r/460475
• 05:40 legoktm: deployed https://gerrit.wikimedia.org/r/460473

2018-09-13

• 19:20 legoktm: deployed https://gerrit.wikimedia.org/r/457109
• 16:22 hashar: Cleaning disk space on deployment-deploy01.deployment-prep.eqiad.wmflabs
• 13:06 hashar: Nuking castor cache /srv/jenkins-workspace/caches/mediawiki-vendor mediawiki/vendor fails composer --ansi validate --no-check-publish
• 08:57 hashar: Updating jenkins job mwselenium-quibble-docker to use Chromium 69/Firefox 60 | T203902
• 08:42 hashar: aborted maintenance-disconnect-full-disks job | T204077
• 05:58 legoktm: updating docker for https://gerrit.wikimedia.org/r/460199
• 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/460197 (no-op)
• 05:13 legoktm: onlined integration-slave-docker-1021
• 05:12 legoktm: legoktm@integration-slave-docker-1021:~$ sudo docker rmi $(sudo docker images -q)
• 05:08 legoktm: deployed
• 02:23 legoktm: deploying https://gerrit.wikimedia.org/r/460183 / https://gerrit.wikimedia.org/r/460184

2018-09-12

• 21:17 marxarelli: adding new jenkins node integration-slave-docker-1034 with 4 executors
• 21:01 marxarelli: launching integration-slave-docker-1034 bigram instance
• 20:59 marxarelli: deleting integration-slave-docker-1003/-1004 instances
• 20:57 marxarelli: taking integration-slave-docker-1003/-1004 offline for replacement
• 20:51 marxarelli: deleting integration-slave-docker-1002 instance
• 20:50 marxarelli: taking integration-slave-docker-1002 offline for replacement
• 20:16 marxarelli: adding newly provisioned integration-slave-docker-1033 jenkins node with 4 executors
• 20:01 marxarelli: launching new integration-slave-docker-1033 bigram instance
• 19:58 marxarelli: replacing integration-slave-docker-1032 offline 85/15% split for docker/workspace left too little space for workspace. puppet change has been updated to use 70/30% volume space ratio
• 18:44 marxarelli: adding jenkins node integration-slave-docker-1032 with 4 executors
• 18:25 marxarelli: launching new bigram instance integration-slave-docker-1032
• 18:21 marxarelli: deleting integration-slave-docker-1027 instance
• 18:20 marxarelli: deleting jenkins node integration-slave-docker-1027 due to insufficient /var/lib/docker space (replaced with 1031 which has dedicated /var/lib/docker volume)
• 18:18 marxarelli: added new jenkins node integration-slave-docker-1031 with 4 executors
• 18:00 marxarelli: launching new xlarge instance integration-slave-docker-1031
• 17:59 marxarelli: deleting integration-slave-docker-1001 instance
• 17:58 marxarelli: deleting now idle node integration-slave-docker-1001
• 17:52 marxarelli: removing integration-slave-docker-1001 jenkins node for replacement with a xlarge instance
• 17:40 marxarelli: adding Jenkins node integration-slave-docker-1030
• 17:36 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/460067/
• 15:49 marxarelli: provisioning new xlarge integration-slave-docker-1030
• 15:47 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459875 on integration-puppetmaster01 for testing
• 09:38 hashar: Updating jobs oojs-ui-docker-publish oojs-ui-npm-run-jenkins-node-6-docker for Chromium 69 and Firefox 60 - T203902
• 00:14 marxarelli: deleting instance integration-slave-docker-1029

2018-09-11

• 22:43 marxarelli: launching m1.xlarge integration-slave-docker-1029 using stretch image
• 22:40 marxarelli: deleting integration-slave-docker-1028 in favor of trying a stretch instance
• 22:05 marxarelli: launching replacement instance integration-slave-docker-1028
• 21:49 marxarelli: removing unresponsive jenkins node integration-slave-docker-1025
• 21:39 marxarelli: cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/459850 on integration-puppetmaster01
• 09:38 hashar: sudo cumin --force 'name:docker' 'rm -fR /srv/jenkins-workspace/workspace/selenium-daily-beta-Wikibase*' # T188742

2018-09-10

• 14:17 hashar: Update CI Quibble jobs to explicitly set the log directory to /log (--log-dir /log) | https://gerrit.wikimedia.org/r/#/c/integration/config/+/458788/
• 08:34 Amir1: ores ae96071 is going to beta

2018-09-09

• 19:47 legoktm: deploying https://gerrit.wikimedia.org/r/459401
• 02:32 legoktm: deploying https://gerrit.wikimedia.org/r/458980
• 00:11 legoktm: updating docker-pkg for https://gerrit.wikimedia.org/r/459265

2018-09-08

• 20:22 legoktm: killed https://integration.wikimedia.org/ci/job/maintenance-disconnect-full-disks/1503/ because it was taking 2+ hours
• 20:20 legoktm: deployed https://gerrit.wikimedia.org/r/457077
• 20:19 legoktm: deleted mwext-PoolCounter-* jobs since they're now unused
• 03:06 legoktm: deploying https://gerrit.wikimedia.org/r/458990 https://gerrit.wikimedia.org/r/458996
• 01:51 legoktm: deploying https://gerrit.wikimedia.org/r/458956 https://gerrit.wikimedia.org/r/458957
• 00:18 legoktm: deployed https://gerrit.wikimedia.org/r/458882

2018-09-07

• 20:01 marxarelli: bringing integration-slave-docker-1006 online again since disk space has been reclaimed
• 19:15 Krinkle: marked integration-slave-docker-1025 offline (no space), aborted builds manualy
• 18:30 legoktm: started gear_client on contint1001
• 18:19 marxarelli: setting integration-slave-docker-1026 executors to 4 to avoid disk space exhaustion due to concurrent builds
• 18:12 Krinkle: Marking integration-slave-docker-1026 offline (ENOSPC)
• 09:30 hashar: integration-slave-docker-1025 lower number of executors from 5 to 4. 8 CPUS can not sustain 5 concurrent Quibble builds | T201972

2018-09-06

• 18:54 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/458553
• 18:48 legoktm: reverted tmpfs change for search-mjolnir-tox-docker
• 17:37 thcipriani: integration-slave-docker-1026 sudo docker kill eae9ba3a1459 -> stopped a container that had been running for 5 hours
• 16:33 thcipriani: mark integration-slave-docker-1026 back online after diskspace recovery
• 12:08 hashar: cleaned integration-slave-docker-1012 and integration-slave-docker-1026

2018-09-05

• 18:31 thcipriani: bring integration-slave-docker-1026 back online since disk space is normal again
• 07:47 elukey: tested and removed a patch to operations/puppet on puppetmaster03. Solved a git rebase conflict between two changes (hope I did it well) and updated the nginx submodule
• 04:08 legoktm: deployed https://gerrit.wikimedia.org/r/c/integration/config/+/457070 (tmpfs for /tmp) to all *tox*docker and *composer*docker jobs
• 03:35 legoktm: deploying https://gerrit.wikimedia.org/r/458095

2018-09-04

• 20:40 Hauskatze: github: created https://github.com/wikimedia/mediawiki-extensions-CreatedPagesList for T203431
• 17:16 legoktm: deploying https://gerrit.wikimedia.org/r/457905
• 04:51 legoktm: deployed https://gerrit.wikimedia.org/r/457808

2018-09-03

• 21:16 legoktm: deploying https://gerrit.wikimedia.org/r/457070 (tmpfs for /tmp) to all *quibble* jobs
• 08:39 hashar: builder releng/quibble-stretch-bundle:0.0.24-4
• 08:21 Hauskatze: github: added "this is a mirror" notice (as nearly all other repos have) to https://github.com/wikimedia/mediawiki-extensions-UserExport
• 07:54 hashar: deployment-deploy01 scap sync-l10n master
• 07:52 hashar: deployment-deploy01 chown -R l10nupdate:l10nupdate /srv/mediawiki-staging/php-master/cache/l10n
• 07:44 hashar: fixing beta-scap-eqiad failure: Operation not permitted: '/srv/mediawiki-staging/php-master/cache/l10n/upstream/l10n_cache-sah.cdb.json'
• 07:05 legoktm: deployed https://gerrit.wikimedia.org/r/457360
• 05:35 legoktm: deployed https://gerrit.wikimedia.org/r/457293
• 05:20 legoktm: building docker images for https://gerrit.wikimedia.org/r/457278

2018-09-02

• 22:25 Hauskatze: maurelio@deployment-sca01:~$ sudo puppet agent -tv | attempting to fix "<shinken-wm> PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]"
• 13:26 Hauskatze: maurelio@deployment-deploy01:/srv/mediawiki-staging/php-master/cache$ sudo -u jenkins-deploy chmod -R 777 l10n/ | more permissions fixes for beta-scap-eqiad
• 12:48 Krenair: deployment-deploy01: `sudo rm /srv/mediawiki/.git/gc.log` to clear error about permissions problems accessing nonexistent file.
• 12:28 Hauskatze: root@deployment-deploy01:/srv/mediawiki-staging# chown -R jenkins-deploy:wikidev portals scap | fixing beta-scap-update-equiad failures

2018-09-01

• 23:52 legoktm: deploying https://gerrit.wikimedia.org/r/457070 for mediawiki-quibble-* jobs
• 22:08 legoktm: deploying https://gerrit.wikimedia.org/r/457068
• 06:53 legoktm: deployed https://gerrit.wikimedia.org/r/456812
• 05:05 legoktm: deployed https://gerrit.wikimedia.org/r/456782
• 02:46 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf mediawiki-core-*lint apps-android-wikipedia-test # /srv was getting full
• 00:40 Krinkle: Disconnect Jenkins slave integration-slave-docker-1026 (ENOSPC)
• 00:02 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase

2018-08-31

• 11:34 hashar: cleaned workspace on integration.integration-slave-jessie-1002
• 06:41 legoktm: deployed https://gerrit.wikimedia.org/r/456547
• 04:26 legoktm: deploying https://gerrit.wikimedia.org/r/456535
• 04:09 legoktm: deploying https://gerrit.wikimedia.org/r/456534
• 03:09 legoktm: deploying https://gerrit.wikimedia.org/r/456531
• 02:42 legoktm: deploying https://gerrit.wikimedia.org/r/456530
• 02:28 legoktm: deployed https://gerrit.wikimedia.org/r/456528

2018-08-30

• 20:47 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/455269 for 18 affected jobs
• 20:34 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticExpressiveness mirror for T199417
• 20:31 Hauskatze: github: deleting https://github.com/wikimedia/mediawiki-extensions-SemanticLinks mirror for T199422
• 20:21 Hauskatze: github: added standard description to https://github.com/wikimedia/mediawiki-extensions-FundraisingEmailUnsubscribe
• 19:38 thcipriani: bring integration-slave-docker-1025 back online
• 19:38 Hauskatze: github: deleted mirror of extension-MwEmbedSupport per T197918
• 16:14 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456362/
• 07:53 legoktm: deploying https://gerrit.wikimedia.org/r/456343
• 07:39 legoktm: deployed https://gerrit.wikimedia.org/r/456341
• 07:19 legoktm: deploying https://gerrit.wikimedia.org/r/456340
• 07:04 legoktm: deploying https://gerrit.wikimedia.org/r/456337
• 05:57 legoktm: manually triggering jobs on contint1001 (this time with 8 parallel jobs, up from 6 previously)
• 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/456328
• 02:55 legoktm: rebuilding docker for https://gerrit.wikimedia.org/r/456073

2018-08-29

• 18:43 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456168/ (T203093)
• 18:10 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/456189 (T203110)
• 17:23 Reedy: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/456180/
• 12:35 hashar: fixed keyholder on deployment-deploy01
• 08:51 hashar: cleaned up disk on integration-slave-docker-1001 and integration-slave-docker-1002

2018-08-28

• 18:55 hasharAway: Bump Jenkins Quibble jobs to 0.0.23 | https://gerrit.wikimedia.org/r/#/c/integration/config/+/455881/ | T203009
• 18:04 hasharAway: Rebuilding Docker quibble containers to use Quibble 0.0.23 | T203009
• 17:30 legoktm: reverted back to previous quibble version (T203009)
• 16:59 hashar: Switched Jenkins jobs to Quibble 0.0.22 | https://gerrit.wikimedia.org/r/455861
• 16:13 hashar: building docker containers for quibble 0.0.22
• 16:06 hashar: tagged quibble 0.0.22 5093492

2018-08-27

• 19:45 legoktm: deployed https://gerrit.wikimedia.org/r/446185 https://gerrit.wikimedia.org/r/455226
• 18:49 Amir1: ores:0e8cc73 is going to beta cluster
• 13:47 hashar: updating phplint jobs to use shallow clone AND wipe the workspace | https://gerrit.wikimedia.org/r/#/c/integration/config/+/405722/ | T179963
• 13:25 hashar: Delete gerrit repo operations/debs/pkg-php/php-ast , was for php-ast 1.6 which is now in Debian | T174338

2018-08-25

• 06:09 legoktm: deployed https://gerrit.wikimedia.org/r/455282 https://gerrit.wikimedia.org/r/455285
• 04:34 legoktm: deployed https://gerrit.wikimedia.org/r/455279
• 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/454776
• 04:23 legoktm: deployed https://gerrit.wikimedia.org/r/455107

2018-08-24

• 19:47 Krenair: set profile::elasticsearch::cirrus::tls_port: 9243 to appease puppet on deployment-elastic* hosts following https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/447568/
• 12:01 addshore: manaully install nmon on integration-slave-docker-1025 to inspect IO
• 11:55 addshore: manaully install iotop on integration-slave-docker-1025 to inspect IO

2018-08-23

• 23:27 Krenair: Fixed puppet rebasing on deployment-puppetmaster03 by removing old cherry-picked version as of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/444610/ - it broke puppet rebasing since Wednesday
• 17:28 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for 270 affected jobs
• 16:43 marxarelli: Deploying https://gerrit.wikimedia.org/r/c/integration/config/+/454706 for mediawiki-quibble-vendor-mysql-php70-docker only
• 13:47 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep again
• 11:06 _joe_: reverting my cherry-pick
• 11:03 _joe_: cherry-picking https://gerrit.wikimedia.org/r/c/operations/puppet/+/453093 on deployment-prep
• 04:42 TimStarling: on deployment-puppetmaster03 running puppet-merge

2018-08-22

• 20:41 mdholloway: deployment-maps04 deployed kartotherian and tilerator to latest
• 20:05 bearND: (beta): Update mobileapps to 141ff20 (T202105 T202237)
• 12:49 Krenair: deleted deployment-deploy01:/tmp/scap_l10n_2097651226 from 31st July to free up space
• 08:08 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-PdfExport T202220
• 06:24 legoktm: deploying https://gerrit.wikimedia.org/r/454471
• 06:20 legoktm: deploying https://gerrit.wikimedia.org/r/454202 plus 3 commits hashar didn't deploy :< (<3)
• 05:37 TimStarling: on deployment-db03 installing pt-heartbeat

2018-08-21

• 18:39 mateusbs17: deployment-maps04 kartotherian/deploy@2047778 "Updating snapshot package on kartotherian"
• 10:37 Amir1: ladsgroup@deployment-deploy01:~$ mwscript extensions/Wikibase/repo/maintenance/rebuildTermSqlIndex.php --wiki=wikidatawiki --rebuild-all-terms (T202260)
• 08:09 hashar: castor: removing cache for mediawiki-extensions-* jobs (no more used) rm -fR /srv/jenkins-workspace/caches/*/*/mediawiki-extensions* | T202341
• 07:46 legoktm: deployed https://gerrit.wikimedia.org/r/454201
• 04:20 legoktm: deployed https://gerrit.wikimedia.org/r/454192
• 04:05 legoktm: deploying https://gerrit.wikimedia.org/r/453975
• 01:13 legoktm: triggering seccheck jobs manually on contint1001

2018-08-20

• 20:37 marxarelli: Reconfigured integration-slave-docker-1026 back to 8 executors (at least 1 vcpu per concurrent job)
• 20:30 marxarelli: Correction: integration-slave-docker-1026 configured w/ 9 executors
• 20:18 marxarelli: Adding jenkins node integration-slave-docker-1026 (bigram instance) w/ 8 executors
• 20:16 mateusbs17: (beta): Update mobileapps to 95e976d (T202105)
• 19:53 marxarelli: Deleting instance integration-slave-docker-1016
• 19:52 marxarelli: Taking integration-slave-docker-1016 node offline for replacement
• 18:24 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/454038/
• 18:19 marxarelli: Deleting integration-slave-docker-1020 instance
• 18:16 marxarelli: Taking integration-slave-docker-1020 node offline for replacement
• 15:58 hashar: deleting Jenkins slave deployment-tin.eqiad the instance has been replaced | T192561
• 02:36 legoktm: deploying https://gerrit.wikimedia.org/r/453899
• 01:42 legoktm: running jobs manually on contint1001 with gear_client
• 01:34 legoktm: deploying https://gerrit.wikimedia.org/r/453895
• 01:24 legoktm: deploying https://gerrit.wikimedia.org/r/453586 https://gerrit.wikimedia.org/r/453894
• 01:08 legoktm: deploying https://gerrit.wikimedia.org/r/452580 https://gerrit.wikimedia.org/r/452617 https://gerrit.wikimedia.org/r/447758
• 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/453419 https://gerrit.wikimedia.org/r/453884

2018-08-18

• 00:45 mdholloway: moved maps-beta.wmflabs.org proxy from deployment-maps03 to deployment-maps04

2018-08-17

• 21:10 marxarelli: removing local docker images that are not tagged 'latest' to free up space on contint1001
• 19:19 marxarelli: pruning stopped containers and dangling images on contint1001 to free up space
• 16:47 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450635/
• 13:23 addshore: reload zuul for: Add WikibaseMediaInfo to Wikibase dependencies [integration/config] - https://gerrit.wikimedia.org/r/453400 (https://phabricator.wikimedia.org/T202066) (owner: Addshore)
• 13:17 addshore: reload zuul for: Use Quibble for WikibaseMediaInfo [integration/config] - https://gerrit.wikimedia.org/r/453396 (https://phabricator.wikimedia.org/T202067) (owner: Addshore)
• 12:32 addshore: reload zuul for: Add extension-quibble template to Wikibase [integration/config] - https://gerrit.wikimedia.org/r/453385 (https://phabricator.wikimedia.org/T200508) (owner: Addshore)
• 08:46 addshore: reloaded zuul
• 04:29 legoktm: deployed https://gerrit.wikimedia.org/r/452619 https://gerrit.wikimedia.org/r/452732 https://gerrit.wikimedia.org/r/452618

2018-08-16

• 14:37 mdholloway: deployed [mobileapps/deploy@166eafa]: Update mobileapps to a808c9d (T201979)

2018-08-15

• 21:21 Krinkle: krinkle@deployment-deploy01: Removing php-master/StartProfiler.php for T201782.
• 16:24 Reedy: beta-update-databases-eqiad broken due to ooui patch not being merged by jerkins

2018-08-14

• 20:23 Krenair: deactivated and cleaned node for deployment-elastic09 which vanished mysteriously
• 17:10 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/452714
• 16:58 mateusbs17: deployment-maps04 kartotherian/deploy@a0f7111 Update node mapnik to 3.7.2 - Fixing submodule deploy
• 16:58 mateusbs17: deployment-maps04 tilerator/deploy@f4c4359 Update node mapnik to 3.7.2 - Fixing submodule deploy
• 16:00 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452634/
• 14:33 mateusbs17: deployment-maps04 kartotherian/deploy@fed001c Update node mapnik to 3.7.2
• 14:33 mateusbs17: deployment-maps04 tilerator/deploy@31e8585 Update node mapnik to 3.7.2

2018-08-13

• {{safesubst:SAL entry|1=23:05 Krenair: deactivated and cleaned puppet node entries for deployment-{tin,mira} T192561}}
• 16:54 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/452361/
• 16:03 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/451247
• 16:01 addshore: reloaded zuul 3 hours ago for https://gerrit.wikimedia.org/r/452356 ......

2018-08-12

• 23:56 legoktm: deploying https://gerrit.wikimedia.org/r/452277
• 23:13 legoktm: deployed https://gerrit.wikimedia.org/r/452276
• 22:43 legoktm: deployed https://gerrit.wikimedia.org/r/444350 https://gerrit.wikimedia.org/r/447119 https://gerrit.wikimedia.org/r/451525
• 22:29 legoktm: deployed https://gerrit.wikimedia.org/r/451059 https://gerrit.wikimedia.org/r/451058
• 22:18 legoktm: deployed https://gerrit.wikimedia.org/r/451319
• 07:48 legoktm: building doxygen image for https://gerrit.wikimedia.org/r/452179

2018-08-10

• 17:58 marxarelli: Reloading zuul to deploy I32007b
• 16:47 marxarelli: Adding "Blubber" label to all integration-slave-docker-* nodes in jenkins

2018-08-09

• 22:19 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451798
• 20:09 Krenair: deleted deployment-urldownloader (the oldest host still running, on trusty, from feb 2015) and replaced with deployment-urldownloader02
• 17:13 bearND: (beta): Update mobileapps to 616ffef
• 01:42 awight: T201518: ORES, fawiki wp10, misc updates

2018-08-08

• 20:39 James_F: Updated mobileapps to 162ebcf in Beta Cluster
• 15:50 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451212
• 00:24 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/451055 / T199850

2018-08-07

• 21:37 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/450157/
• 19:59 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450435 / T201107)
• 19:52 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/450644
• 19:35 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/451005

2018-08-06

• 16:11 addshore: reload zuul for https://gerrit.wikimedia.org/r/450604
• 15:42 addshore: reload zuul for https://gerrit.wikimedia.org/r/450590
• 07:53 addshore: reload zuul (I think, without using fab) for https://gerrit.wikimedia.org/r/450514

2018-08-04

• 02:40 legoktm: deployed https://gerrit.wikimedia.org/r/450296

2018-08-02

• 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/450149
• 20:12 legoktm: deployed https://gerrit.wikimedia.org/r/450085
• 19:01 legoktm: removed all docker images from integration-slave-docker-1004 to free up root partition, cleaned up all workspaces while I was at it (T201077)
• 16:27 gehel: re-imaging depoyment-elastic* to stretch completed
• 14:07 gehel: re-imaging depoyment-elastic* to stretch
• 07:47 legoktm: triggering jobs directly on contint1001 w/ gear_client.py

2018-08-01

• 23:00 legoktm: deploying https://gerrit.wikimedia.org/r/449898 https://gerrit.wikimedia.org/r/c/integration/config/+/447454
• 14:56 hashar: Phasing out mediawiki-extensions jobs in favor of wmf-quibble ones | T197469
• 13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 (T199801)
• 10:08 dcausse: deployment-prep T199270: fix dirty diff on wmf-config/event-schema submodule on deployment-tin and deployment-deploy01

2018-07-31

• 22:04 thcipriani: moving active deployment-prep deployment server to deployment-deploy01
• 19:50 marxarelli: Configuring Jenkins to include integration/pipelinelib as an available pipeline library
• 17:50 Krinkle: Apply role::webperf::profiling_tools to deployment-webperf12; T195312 / T180761
• 10:15 hashar: gerrit: deleting branch wmf/es6 on mediawiki/vendor . We use 'es6' branch instead. (made for dcausse )
• 06:05 legoktm: deployed https://gerrit.wikimedia.org/r/449402
• 05:56 legoktm: deploying https://gerrit.wikimedia.org/r/449399

2018-07-30

• 23:28 Krinkle: Setting up puppet cert for deployment-webperf12; T195312 / T180761
• 23:06 Krinkle: Perform node deactivate/clean for deleted deployment-webperf13
• 23:04 Krinkle: Create instance deployment-webperf13 (deployment-webperf13 ) - T195312 / T180761
• 23:03 Krinkle: Delete and recreate deployment-webperf13 (T195312 / T180761)
• 21:38 Krenair: ran package upgrades on deployment-imagescaler02 to fix thumbor
• 20:57 mdholloway: deployment-maps04 restarted postgresql to pick up the correct data_directory value (T200634)
• 19:44 hasharAway: renaming Jenkins job "mediawiki-core-code-coverage" to "mediawiki-core-code-coverage-docker" to keep history. It is migrated by https://gerrit.wikimedia.org/r/#/c/integration/config/+/449273/ | T195918
• 19:34 Krenair: copied /etc/pam.d/sshd from deployment-restbase01 to deployment-restbase02 to fix missing MOTD, T195709
• 17:04 hashar: github: granted @thcipriani owner rights
• 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/449035 https://gerrit.wikimedia.org/r/449107 https://gerrit.wikimedia.org/r/449087

2018-07-29

• 05:34 legoktm: deployed https://gerrit.wikimedia.org/r/449002

2018-07-28

• 22:08 legoktm: deployed https://gerrit.wikimedia.org/r/448858
• 03:24 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover-extensions$ sudo -u jenkins-slave rm -rf EtherEditor/ # T197698

2018-07-27

• 21:11 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/448762/
• 14:58 hashar: building releng/quibble-stretch:0.0.21-1 container | T195918
• 14:43 hashar: Building tox-certcentral container for vgutierrez !
• 13:13 hashar: Refreshing Quibble Jenkins jobs to use 0.0.21 | https://gerrit.wikimedia.org/r/#/c/integration/config/+/448479/
• 12:05 hashar: Building releng/quibble images 0.0.21
• 11:40 hashar: Tagged quibble 0.0.21 e681dce021e
• 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/ T200508
• 11:38 addshore: reload zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/448468/
• 01:11 legoktm: deployed https://gerrit.wikimedia.org/r/443433 https://gerrit.wikimedia.org/r/447337

2018-07-26

• 16:08 legoktm: deployed https://gerrit.wikimedia.org/r/447227
• 15:03 zeljkof: Reloading Zuul to deploy 208a505
• 04:00 kart_: Updated cxserver to e98c81b

2018-07-24

• 05:22 kart_: Updated cxserver to d3c9d15

2018-07-23

• 20:06 mdholloway: deployment-mcs01 deployed [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5
• 19:16 mdholloway: deployment-maps04 deployed dependency updates for agreement on mapnik v3.5.14
• 16:25 Krinkle: Applying role::webperf::profiling_tools class to deployment-webperf13, T195312
• 16:04 Krinkle: Set up puppet cert stuff on deployment-webperf13 T195312
• 15:57 Krinkle: Set 'puppetmaster' Hiera for deployment-webperf13 / T195312
• 15:34 Krinkle: Creating deployment-webperf13 - T195312
• 15:34 Krinkle: Deleting deployment-webperf12 - T195312
• 10:26 hashar: Regenerate all debian-glue jobs from JJB. Just to be sure.
• 10:24 hashar: Regenerate https://integration.wikimedia.org/ci/job/debian-glue/ from JJB. The timeout is forced to 3 when it should rely on the BUILD_TIMEOUT environment variable

2018-07-19

• 21:47 hashar: CI: Force npm loglevel to 'info' via a Zuul parameter | T198348
• 21:27 hashar: nodepool: deleted a few lost instances due to jenkins arbitrary reboots
• 15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes | https://gerrit.wikimedia.org/r/#/c/446863 | T198348
• 08:52 hashar: Switching back timeout of Quibble jobs from 30 to 45 minutes | https://gerrit.wikimedia.org/r/442989 | T198348

2018-07-17

• 01:36 Krinkle: Applying role::webperf::profiling_tools class to webperf12 in Beta Cluster - T195312, T180761.

2018-07-16

• 21:10 awight: ran namespaceDupes.php on beta enwiki
• 19:56 bearND: (beta): Update mobileapps to bed7b29

2018-07-14

• 21:20 Krenair: redirected security_audit traffic (see T72181) traffic from deployment-mediawiki06 to deployment-mediawiki-09 to fix puppet on varnish (06 was deleted in T192996)
• 03:27 Krinkle: Clearing various workspaces on integration-slave-jessie-1001 to fix operations-mw-config-php55lint Jenkins builds - T179963

2018-07-13

• 17:31 Reedy: Reloading Zuul to deploy stuff

2018-07-12

• 22:45 Krenair: deployment-maps04 groupadd -g 1000 maps-admins
• 19:04 mdholloway: deployment-prep: deleting new instance deployment-maps04 (initial puppet run failed) and creating deployment-maps05

2018-07-11

• 20:07 bearND: (beta): Update mobileapps to b5e152d
• 19:53 Reedy: next set of db updates for beta might be a bit slow. Expected!
• 13:35 mdholloway: deployment-prep: launched new instance deployment-maps04 for maps testing on stretch

2018-07-10

• 21:48 hashar: manually cleaning up disk space on integration-slave-jessie-1002 ( apt-get clean , apt-get autoremove --purge , nuked /srv/jenkins-workspace/workspace/*lint
• 21:27 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-Transliterator | T199103
• 18:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443006 (and apparently https://gerrit.wikimedia.org/r/444884 as well)
• 18:31 Krenair: deleted deployment-dumps-puppetmaster earlier per ariel in -releng (some details in T72792)
• 13:05 hashar: Building container releng/quibble-stretch:0.0.20 https://gerrit.wikimedia.org/r/444876 | T198171

2018-07-09

• 23:24 Krinkle: Ran `puppet node deactivate` and `puppet node clean` for deployment-apertium02.deployment-prep.eqiad.wmflabs (T142152)
• 22:09 Krinkle: Deleting instance deployment-apertium02 - T142152
• 19:10 Krenair: deleted deployment-mediawiki06 T192996
• 17:35 zeljkof: Reloading Zuul to deploy 7967dad
• 10:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-Premoderation | T198952
• 10:47 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-SwarmExport | T198957
• 10:41 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-VisualChat | T198955
• 10:40 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-ContributorsAddon | T198864

2018-07-08

• 20:17 Krinkle: Shutting off deployment-apertium02 (T142152)
• 18:31 Krenair: deleted deployment-mx T184244
• 16:55 Krenair: deleted deployment-redis01 T179371
• 16:54 Krenair: deleted deployment-redis02 T179371
• 16:42 Krenair: deleted deployment-puppetmaster02. root and home files are in archives under deployment-puppetmaster03:/root/

2018-07-07

• 03:38 thcipriani: deployment-tin:sudo rm -rf /srv/mediawiki/.git

2018-07-06

• 03:53 kart_: Updated cxserver to bfc9c84

2018-07-05

• 06:27 kart_: Updated cxserver to f8c71a1

2018-07-04

• 20:15 Reedy: beta unbroke beta code autodeploy T72597
• 05:31 Krinkle: Fix beta-scap-eqiad by cherry-picking https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/443767/ T192996
• 05:31 Krinkle: beta-scap-eqiad failing due to mediawiki06 still being a scap target (despite being shutdown per T192996)

2018-07-03

• 22:11 Krenair: rebased stuff on deployment-puppetmaster and ran puppet everywhere through cumin
• 21:48 Krenair: cleaned up cherry-pick conflict to try to fix puppet
• 08:44 hashar: Building docker container releng/tox-labs-striker to add libssl-dev | T198076
• 04:31 Krinkle: Setting up puppetmaster/cerf for deployment-webperf12 (T195312)
• 03:49 Krinkle: Create deployment-webperf12 as equivalent of webperf1002/webperf2002 in prod (T195312, T194390)
• 03:41 Krinkle: Shut off deployment-mediawiki06 from Horizon (but not yet deleted) - T192996

2018-07-02

• 20:28 hashar: building docker image releng/tox:0.1.1 , that adds libssl-dev | T198076
• 16:35 hashar: github: delete wikimedia/mediawiki-extensions-PubSubHubbub | T198322
• 00:25 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/443023

2018-06-30

• 20:40 Krenair: ran git gc on deployment-tin:/srv/mediawiki to free up space

2018-06-28

• 21:17 hasharAway: castor02: nuking cache of npm/node jobs via rm -fR /srv/jenkins-workspace/caches/*/*/*node* (note: other jobs might still have a npm cache) | T198348
• 21:09 hasharAway: castor: nuking caches castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-hhvm-docker/npm and castor-mw-ext-and-skins/master/wmf-quibble-vendor-mysql-php70-docker/npm | T198348
• 21:07 hasharAway: castor: nuking caches mediawiki-core/master/mediawiki-quibble-vendor-mysql-php70-docker and mediawiki-core/master/mediawiki-quibble-vendor-mysql-hhvm-docker | T198348
• 11:18 Hauskatze: Ran namespaceDupes for eswiki and eswikibooks following namespace changes on JADE.

2018-06-27

• 16:27 hashar: Building Docker containers releng/quibble-jessie-php55:0.0.19-1 and releng/quibble-stretch:0.0.19-1 | T196346 T198336
• 04:45 mdholloway: deployed to beta: [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba

2018-06-26

• 22:17 Krenair: arming keyholder on deployment-deploy01
• 20:16 Krenair: done the same on -sca02 and -apertium02
• 20:09 Krenair: package upgrades on -sca01 to try to fix apertium stuff
• 15:32 hashar: repooling integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 (converted to m1.medium instances)
• 15:11 hashar: Deleting integration-slave-docker-1013 integration-slave-docker-1014 and integration-slave-docker-1015 . Recreating them as m1.medium instances
• 15:11 hashar: Changing integration-slave-docker-1012 in jenkins m1executor -> m4executor. It is a m1.medium instance and can thus run mediawiki jobs
• 13:18 hashar: cleaned containers on integration-slave-docker-1006

2018-06-25

• 21:19 mdholloway: deployed to beta: [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52
• 17:45 Krenair: fixed puppet repo rebasing
• 16:20 hashar: deployment-prep: git gc on a few repositories under /srv/mediawiki-staging/php-master
• 12:19 kart_: "Beta: Update cxserver to ece5e7a"
• 07:49 hashar: github: deleting archived repository https://github.com/wikimedia/mediawiki-extensions-CommunityVoice | T196618

2018-06-23

• 19:19 Krenair: restarted hhvm on -mediawiki-07 then apache2 to bring beta back up

2018-06-22

• 21:37 hashar: deployment-prep: sudo cumin --force '*' 'apt-get clean' # some instances are causing disk warnings
• 14:46 zeljkof: Reloading Zuul to deploy 94c38c1
• 12:34 hashar: Building docker images for Quibble 0.0.19 | T197687
• 12:10 hashar: Tagging quibble 0.0.19 which pass "--autoplay-policy=no-user-gesture-required" to Chromium | T197687

2018-06-21

• 12:04 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EtherEditor | T197698
• 10:40 hashar: Buidling docker image releng/operations-puppet:0.3.2 https://gerrit.wikimedia.org/r/c/integration/config/+/441349/

2018-06-20

• 19:47 hasharAway: Removing old Quibble images from the CI Docker slaves
• 17:33 hashar: Refreshing Quibble Jenkins jobs to use 0.0.18-2 docker image. That adds JSDuck | T197806
• 16:32 hashar: Rebuilding Quibble images to get jsduck included in the image | T197806

2018-06-19

• 15:01 hashar: CI docker slaves: docker container prune --force ; docker image prune --force
• 05:12 Amir1: ladsgroup@deployment-tin:~$ foreachwikiindblist all-labs deleteAutoPatrolLogs.php

2018-06-18

• 17:25 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
• 16:15 Amir1: ladsgroup@deployment-tin:~$ mwscript populateChangeTagDef.php --wiki=enwiki
• 16:07 Amir1: change_tag_def is back now
• 16:05 Amir1: making enwiki in beta clsuter readonly

2018-06-15

• 12:19 zeljkof: Reloading Zuul to deploy 529965b
• 08:30 hashar: github: deleted lot of archived/read-only extensions and skins | T180864
• 07:54 hashar: apt-get upgrade on integration-slave-jessie-android
• 07:54 hashar: Fixing puppet on integration-slave-jessie-android . Has been stall for a month due to jenkins-debian-glue-buildenv.deb which could not be upgraded
• 07:51 hashar: cleaning disk space on integration-slave-jessie-1001

2018-06-14

• 21:47 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/439572/
• 14:26 zeljkof: Reloading Zuul to deploy 04b1c86
• 08:45 Hauskatze: maurelio@deployment-tin:~$ mwscript namespaceDupes.php --wiki=eswiki --fix (64 broken pagelinks resolved, no remaining conflicts)
• 07:40 hashar: Cleaned up bunch of Docker containers and images from the CI slave-docker-* instances
• 07:40 hashar: Armed keyholder on integration-cumin using passphrase from integration-puppetmaster01| T197207

2018-06-13

• 19:47 Krenair: powered off deployment-mx T184244
• 07:21 hashar: github: deleting archived repo wikimedia/operations-software-tessera | TT186096
• 05:55 legoktm: deployed https://gerrit.wikimedia.org/r/440054
• 01:10 legoktm: deploying https://gerrit.wikimedia.org/r/440027

2018-06-12

• 17:40 halfak: deploying ores 36037b6
• 01:53 legoktm: deployed https://gerrit.wikimedia.org/r/439829

2018-06-11

• 21:47 halfak: deploying ores 6ee8775
• 18:13 legoktm: deployed https://gerrit.wikimedia.org/r/436957 https://gerrit.wikimedia.org/r/439547
• 11:59 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-DidYouMean | T196430
• 00:12 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2018-06-10

• 17:03 Krenair: also fixed scap install on -zotero01
• 04:09 Krenair: added SPF and DMARC records to beta.wmflabs.org
• 00:42 Krenair: cumin 'P{R:Package = scap} and P{F:lsbdistcodename = jessie}' 'apt-get install scap=3.8.2-1+0~20180607230422.353~1.gbp2bb4cc -y --force-yes'

2018-06-09

• 21:00 Krenair: Temporarily substituting certificates on deployment-cache-text04 for certs generated from T182927 to test
• 02:20 Krenair: stopping deployment-puppetmaster02 again, looks like it was automatically booted by novaadmin after security patches a couple days ago
• 02:17 Krenair: shut down old deployment-dumps-puppetmaster instance (replaced with a newer stretch instance), emailed ariel
• 02:13 Krenair: shut down old deployment-redis01 and deployment-redis02 instances T179371

2018-06-08

• 18:05 legoktm: updating docker files for https://gerrit.wikimedia.org/r/438152
• 09:48 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-EditOwn | T196433
• 07:50 hashar: github: deleting archived repo https://github.com/wikimedia/mediawiki-extensions-InterwikiIntegration | T196442
• 07:46 hashar: nuking integration-slave-jessie-1002 jenkins workspaces

2018-06-07

• 23:13 Krenair: armed keyholder on deployment-cumin after reboot, found passphrase at deployment-puppetmaster03:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase
• 18:51 Krenair: deployment-deploy-01 rejecting all connections, rebooting
• 17:24 Reedy: unstuck jenkins deployments to beta
• 14:27 _joe_: force-run puppet on deployment-prep appservers, restarted nutcracker after reconfigruation
• 14:16 _joe_: restarted apache2 in deployment-puppetmaster02 in deployment-prep, which is the correct way to run the puppetmaster there
• 14:16 _joe_: killed "puppet master" process in depoloyment-prep
• 14:03 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-WikivoteMapsYandex | T193844
• 11:22 hashar: Switching quibble jenkins jobs to 0.0.18 images
• 10:41 hashar: Building Docker images for Quibble 0.0.18
• 09:15 addshore: added wmde-leszek to deployment-prep

2018-06-06

• 21:40 bearND: BC: Update mobileapps to 5ea008c
• 20:36 Amir1: deploying 65ce165 on ores
• 20:25 Amir1: deploying 65e979f on ores
• 19:59 bearND: Update mobileapps to 3bf9be5 on BC
• 19:47 hashar: github: delete https://github.com/wikimedia/mediawiki-extensions-Genderize (archived repository) | T196108
• 19:14 hashar: github: deleted https://github.com/wikimedia/mediawiki-extensions-SolrStore (archived repo) | T196513
• 18:58 thcipriani: @deployment-tin: sudo keyholder arm
• 17:09 hashar: github: deleting https://github.com/wikimedia/mediawiki-extensions-CategorySlideShow | T186969
• 17:03 hashar: Phabricator: marked SkelJS Diffusion repository inactive https://phabricator.wikimedia.org/diffusion/ESJS/ | T196509
• 17:01 hashar: github: deleted archived repository SkelJS https://github.com/wikimedia/mediawiki-extensions-SkelJS | T196509

2018-06-05

• 19:59 mdholloway: deployed to BC: [mobileapps/deploy@7ecc3b6]: Update mobileapps to 66727b7
• 12:07 kart_: Update cxserver to 7fb7671

2018-06-04

• 20:37 mdholloway: Deployed to BC: [mobileapps/deploy@276ea43]: Update mobileapps to f579f0d
• 19:37 Reedy: running foreachwiki maintenance/deduplicateArchiveRevId.php --force T196401
• 15:44 awight: ORES: Fix T194322

2018-06-02

• 21:20 legoktm: legoktm@integration-slave-docker-1003:~$ sudo docker rmi $(sudo docker images -q)
• 19:57 greg-g: gjg@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *
• 19:31 Krenair: restarted parsoid on deployment-parsoid09 to try to fix stuff
• 18:07 Krinkle: Beta Cluster's RESTBase or Parsoid is broken. Saving VE times out, logstash-beta contain restbase: "internal_http_error" / "Error: ESOCKETTIMEDOUT"
• 01:16 legoktm: running docker-pkg in a screen because my connection is super flaky

2018-06-01

• 22:17 Reedy: https://gerrit.wikimedia.org/r/#/c/436902/ finished deploying
• 21:37 Krinkle: Re-create performance-beta.wmflabs.org webproxy (wired to webperf01) - T195314
• 21:29 Krinkle: Re-creating webperf01 in deploymet-prep, T195314
• 20:57 legoktm: deploying docker-pkg with https://gerrit.wikimedia.org/r/436859 for reals this time (again)
• 20:51 legoktm: deleting old versions of docker images
• 20:48 hashar: contint1001: deleting some old wikimedia/mediawiki-services-mathoid docker images
• 20:40 mutante: contint1001 - mkdir /srv/zuul-debug-logs ; mv debug.log.2018-05-* from /var/log/zuul/ over there to free up disk space on / VG
• 20:26 mutante: contint1001 - apt-get clean got a little bit more disk space
• 20:07 legoktm: really Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
• 20:02 Reedy: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/436852
• 13:52 hashar: Tagged Quibble 0.0.17, rebuilding Docker images and bumping jenkins jobs
• 08:45 hashar: Bumping Quibble jobs to 0.0.16
• 02:39 legoktm: deployed https://gerrit.wikimedia.org/r/436716

2018-05-31

• 23:55 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/436701
• 18:13 hashar: deployment-prep integration : revert puppet change https://gerrit.wikimedia.org/r/#/c/436600/ | T187622
• 18:04 Krinkle: un-cherry-pick https://gerrit.wikimedia.org/r/#/c/436581/
• 18:03 Krinkle: removed scap::sources override from Horizon Puppet config for deployment-prep - T195314
• 11:41 hashar: Fixed EXT_DEPENDENCIES no more being injected on quibble jobs T196062
• 08:43 legoktm: deployed https://gerrit.wikimedia.org/r/436482
• 08:21 legoktm: deployed https://gerrit.wikimedia.org/r/436403
• 07:54 legoktm: deployed https://gerrit.wikimedia.org/r/436477
• 07:42 legoktm: deployed https://gerrit.wikimedia.org/r/436471
• 07:16 legoktm: repooled integration-slave-docker-1010
• 06:41 legoktm: depooled integration-slave-docker-1010 in jenkins for testing
• 01:19 Krinkle: Add web proxy in Horizon for 'performance-beta', mapping to deployment-webperf01 port 80 - T195314
• 00:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/389412
• 00:40 Krinkle: Apply 'role::webperf' to deployment-webperf01, T195314
• 00:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434722

2018-05-30

• 21:07 legoktm: deployed https://gerrit.wikimedia.org/r/436404, reverted quibble upgrade
• 20:37 legoktm: deployed https://gerrit.wikimedia.org/r/436351
• 20:28 hasharAway: contint1001: triggered a few quibble runs from contint1001. Running in a screen
• 20:27 hasharAway: contint1001: (for ext in BlueSpiceAbout BlueSpiceArticleInfo BlueSpiceAuthors BlueSpiceAvatars BlueSpiceBlog BlueSpiceCategoryManager BlueSpiceChecklist BlueSpiceConfigManager BlueSpiceContextMenu BlueSpiceCountThings BlueSpiceEditNotifyConnector BlueSpiceEmoticons BlueSpiceExtendedFilelist BlueSpiceExtendedSearch BlueSpiceExtendedStatistics BlueSpiceExtensions BlueSpiceFoundation BlueSpiceGroupManager BlueSpiceHideTitle
• 18:32 mutante: created instance deployment-deploy-01 with stretch and flavor x-large (T192561)

2018-05-29

• 23:41 legoktm: deployed https://gerrit.wikimedia.org/r/436190
• 18:21 legoktm: deployed https://gerrit.wikimedia.org/r/436020
• 14:25 gehel: rolling restart of elasticsearch deployment-prep for plugin upgrade - T193734
• 12:42 zeljkof: Reloading Zuul to deploy bcd36d9
• 07:36 hashar: cleaning up some caches on castor02.integration.eqiad.wmflabs (under /srv/jenkins-workspace/caches/ )
• 06:12 legoktm: deployed https://gerrit.wikimedia.org/r/435961
• 04:47 legoktm: building https://gerrit.wikimedia.org/r/430037
• 01:57 legoktm: deployed https://gerrit.wikimedia.org/r/435952
• 01:34 legoktm: deployed https://gerrit.wikimedia.org/r/435950
• 01:31 legoktm: deployed https://gerrit.wikimedia.org/r/435949
• 01:27 legoktm: deployed https://gerrit.wikimedia.org/r/435948
• 01:16 legoktm: deployed https://gerrit.wikimedia.org/r/435706 https://gerrit.wikimedia.org/r/435707 https://gerrit.wikimedia.org/r/434799
• 01:02 legoktm: deployed https://gerrit.wikimedia.org/r/435946
• 01:00 legoktm: deployed https://gerrit.wikimedia.org/r/435945
• 00:44 legoktm: deployed https://gerrit.wikimedia.org/r/435943
• 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/435939
• 00:11 legoktm: deployed https://gerrit.wikimedia.org/r/435938

2018-05-28

• 11:09 hashar: Building Quibble Docker images 0.0.13 | T195634
• 11:01 hashar: tagging quibble 0.0.13 hhvm server should set .svg Content-Type | T195634
• 04:47 legoktm: running modified version of hashar's gear_client.py on contint1001, feel free to kill if it causes problems

2018-05-27

• 06:02 legoktm: deployed https://gerrit.wikimedia.org/r/435681
• 00:24 legoktm: deployed https://gerrit.wikimedia.org/r/435673

2018-05-26

• 23:09 Krinkle: Killed a bunch of stuck beta-mediawiki-config-update-eqiad jobs in Jenkins
• 23:06 Krinkle: beta-mediawiki-config-update-eqiad jobs have been stuck in Zuul for 17 hours

2018-05-25

• 23:08 legoktm: deployed https://gerrit.wikimedia.org/r/435278 https://gerrit.wikimedia.org/r/433197 https://gerrit.wikimedia.org/r/434950 https://gerrit.wikimedia.org/r/434727 https://gerrit.wikimedia.org/r/435221
• 21:41 hashar: cleaned up disk space on deployment-tin . Reenabled jenkins slave.

2018-05-24

• 19:12 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434990
• 19:04 legoktm: deleted https://integration.wikimedia.org/ci/view/Labs/job/labs-tools-ZppixBot-php55lint/
• 19:03 legoktm: deployed https://gerrit.wikimedia.org/r/434982 https://gerrit.wikimedia.org/r/434983 https://gerrit.wikimedia.org/r/434985 https://gerrit.wikimedia.org/r/434986
• 18:47 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434980
• 17:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434966 https://gerrit.wikimedia.org/r/434954
• 16:57 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434956
• 15:56 zeljkof: Reloading Zuul to deploy 2e02d97
• 15:13 Reedy: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/434922
• 14:17 zeljkof: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/434917/
• 03:49 legoktm: deployed https://gerrit.wikimedia.org/r/434846

2018-05-23

• 22:24 legoktm: deploying https://gerrit.wikimedia.org/r/434815
• 20:07 bearND: Beta Cluster: Update mobileapps to 29ebe0f
• 14:22 maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force (completed: successfully loaded 941 exit nodes on all Beta Cluster wikis)
• 05:39 legoktm: deployed https://gerrit.wikimedia.org/r/434626
• 05:21 legoktm: deployed https://gerrit.wikimedia.org/r/433960 https://gerrit.wikimedia.org/r/433363

2018-05-22

• 20:42 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo puppet agent -tv; done (for T194926, finished without errors on any of the hosts)
• 20:38 eddiegp: eddie@eddie-thinkpad:~$ for host in deployment-elastic07 deployment-ircd deployment-logstash2 deployment-mathoid deployment-ms-fe02 deployment-prometheus01 deployment-tin; do ssh $host.eqiad.wmflabs sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils; done T194926
• 20:15 eddiegp: eddie@deployment-db03:~$ sudo apt-get -q -y --force-yes -o DPkg::Options::=--force-confold install ldap-utils
• 16:53 Krinkle: Created deployment-webperf01 instance (m1.small) - ref T195312

2018-05-21

• 16:06 Hauskatze: Added `User:Kullfar` to 'confirmed' for two weeks for OAuth-consumer registration, per request from Lucas Werkmeister; cfr. <https://meta.wikimedia.beta.wmflabs.org/w/index.php?oldid=7031#Admin_request>.

2018-05-20

• 16:55 addshore: reload zuul for (Merged) jenkins-bot: Add quibble for Wikibase experimental [integration/config] - https://gerrit.wikimedia.org/r/434198 (owner: Addshore)
• 09:18 greg-g: gjg@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2018-05-19

• 14:56 zeljkof: Reloading Zuul to deploy 9f3a185
• 10:56 Krenair: amended uncommitted changes into HEAD commit (notified author) so I can unbreak puppet updates, also removed my old POC secure redirect puppet patch
• 05:07 legoktm: deployed https://gerrit.wikimedia.org/r/433918
• 04:55 legoktm: deployed https://gerrit.wikimedia.org/r/433911
• 04:48 legoktm: deployed https://gerrit.wikimedia.org/r/433845

2018-05-18

• 18:31 zeljkof: Reloading Zuul to deploy f8e9625
• 01:26 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050

2018-05-17

• 07:18 legoktm: deploying mediawiki-phan-seccheck:0.2.1 image

2018-05-16

• 17:51 twentyafterfour: deployed mobileapps-periodic-test to jenkins with jenkins-job-builder. refs T177896
• 07:09 thcipriani: removed shadow mwdeploy users on deployment-mediawiki-07
• 02:32 legoktm: deployed https://gerrit.wikimedia.org/r/433308

2018-05-15

• 23:39 legoktm: deployed https://gerrit.wikimedia.org/r/433297
• 23:22 legoktm: live hacking zuul config on contint1001
• 05:29 legoktm: deployed https://gerrit.wikimedia.org/r/432947 https://gerrit.wikimedia.org/r/433017 https://gerrit.wikimedia.org/r/433020

2018-05-14

• 19:41 bearND: BC: Update mobileapps to 39c16e4
• 17:16 legoktm: deployed https://gerrit.wikimedia.org/r/432764
• 03:40 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.1
• 03:17 legoktm: Building image docker-registry.discovery.wmnet/releng/wikimedia-audit-resources:0.1.0

2018-05-11

• 00:08 Krinkle: Add Console Section patterns to Jenkins Config for "Selenium"
• 00:08 Krinkle: Update Console Sections in Jenkins Config to collapse setup-* sections by default

2018-05-10

• 22:59 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/432165/
• 22:07 Krinkle: Update Console Section patterns to Jenkins Config for "setup-composer" (didn't match anymore)
• 22:03 Krinkle: Update Console Section patterns to Jenkins Config for "QUnit"
• 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "JS Documentation"
• 22:02 Krinkle: Add Console Section patterns to Jenkins Config for "setup-docker"
• 21:58 MaxSem: Touched everything in beta cluster's wmf-config to fix some files not being synced
• 20:47 duploktm: deployed https://gerrit.wikimedia.org/r/432526
• 20:15 legoktm: deployed https://gerrit.wikimedia.org/r/432051 https://gerrit.wikimedia.org/r/431806 https://gerrit.wikimedia.org/r/431853 https://gerrit.wikimedia.org/r/431610 https://gerrit.wikimedia.org/r/431554 https://gerrit.wikimedia.org/r/432426
• 19:36 legoktm: kicked postmerge queue in jenkins
• 19:10 twentyafterfour: reset jenkins slave on deployment-tin
• 13:19 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy' | T194376
• 12:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Hosiryuhosi' 'Rxy'

2018-05-09

• 22:53 awight: ORES: wheels fixups
• 21:31 awight: Bump ORES wheels
• 21:21 twentyafterfour: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/432150/
• 21:04 awight: ORES: drafttopic in beta
• 16:17 thcipriani: deployment-tin:apt-get clean && rm -rf /srv/mediawiki/.git to free space

2018-05-08

• 19:49 legoktm: aborted mwext-phpunit-coverage-patch #2819
• 12:19 moritzm: upgrading app servers in beta to wikidiff 1.6.0 (T190717)

2018-05-07

• 20:19 bearND: Update mobileapps to c1f4de6 in BC
• 16:30 greg-g: doing the disconnect/reconnect dance as a group in our team meeting: https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update (tyler's doing it)
• 04:01 legoktm: deployed https://gerrit.wikimedia.org/r/431503
• 03:21 legoktm: deployed https://gerrit.wikimedia.org/r/431500 https://gerrit.wikimedia.org/r/431501
• 01:30 legoktm: deployed https://gerrit.wikimedia.org/r/431494 https://gerrit.wikimedia.org/r/431495

2018-05-06

• 20:59 legoktm: deployed https://gerrit.wikimedia.org/r/431308
• 20:39 legoktm: deployed https://gerrit.wikimedia.org/r/431339
• 04:26 legoktm: deployed https://gerrit.wikimedia.org/r/430290

2018-05-05

• afk: revert https://phabricator.wikimedia.org/D1047 from scap to fix beta-scap-eqiad (hopefully)
• 02:27 legoktm: deployed https://gerrit.wikimedia.org/r/431003
• 01:36 legoktm: deploying https://gerrit.wikimedia.org/r/430287
• 01:26 legoktm: updating docker for https://gerrit.wikimedia.org/r/431055

2018-05-04

• 18:55 legoktm: deployed https://gerrit.wikimedia.org/r/430946 https://gerrit.wikimedia.org/r/428068
• 12:59 hashar: ssh deployment-tin.deployment-prep.eqiad.wmflabs sudo rm /var/lock/scap.operations_mediawiki-config.lock

2018-05-03

• 19:20 hashar: Updating Jenkins jobs for quibble-stretch:0.0.11-3 | T193767
• 14:45 greg-g: gjg@deployment-tin:~$ sudo rm /var/lock/scap.operations_mediawiki-config.lock
• 04:09 legoktm: deployed https://gerrit.wikimedia.org/r/430536 https://gerrit.wikimedia.org/r/430537 https://gerrit.wikimedia.org/r/400159

2018-05-02

• 17:42 greg-g: gjg@integration-slave-jessie-1002:/srv/jenkins-workspace/workspace$ sudo rm -rf *
• 17:19 greg-g: gjg@deployment-tin:/var/lock$ sudo rm scap.operations_mediawiki-config.lock
• 16:25 hasharAway: cleaning up disk on integration-slave-docker-1006
• 10:52 hashar: gerrit: added Hauskatze / MarcoAurelio to group "mediawiki-skins-Synagonism" ( https://gerrit.wikimedia.org/r/#/admin/groups/838,members ) for T193611
• 10:43 hashar: Building quibble docker iamges 0.0.11-2 T118371 T193222
• 07:04 legoktm: deployed https://gerrit.wikimedia.org/r/430295
• 06:52 legoktm: building/deploying docker-registry.discovery.wmnet/releng/ci-src-setup:0.2.5 https://gerrit.wikimedia.org/r/430292
• 06:49 legoktm: deployed https://gerrit.wikimedia.org/r/430291

2018-05-01

• 16:51 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=deploymentwiki --logwiki=deploymentwiki 'Spambot~BobFeliciano' 'BobFeliciano'

2018-04-30

• 09:09 hashar: Quibble announce on Phame blog thing https://phabricator.wikimedia.org/phame/post/view/99/introducing_quibble/ (thanks zeljkof )
• 08:32 legoktm: deployed https://gerrit.wikimedia.org/r/429638 https://gerrit.wikimedia.org/r/429740

2018-04-27

• 22:07 hashar: Running quibble-vendor-mysql-php70-docker against ~ 900 MediaWiki extensions. Triggered with a custom gear-client.py script from contint1001. PID 29710
• 21:31 hashar: Running quibble-vendor-mysql-php70-docker against 43 mediawiki skins. Triggered with a custom gear_client.py script from contint1001 . PID 13134

2018-04-26

• 20:19 hashar: bumped quibble Jenkins jobs to 0.0.11
• 19:00 hashar: Building releng/quibble 0.0.11 docker images
• 17:41 mobrovac: stopped cpjobqueue and purging logs
• 16:51 thcipriani: mark integration-slave-jessie-1001 online
• 16:51 thcipriani: thcipriani@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
• 16:49 greg-g: tyler marked integration-slave-jessie-1001 as offline

2018-04-25

• 21:27 hasharAway: building releng/quibble docker images @0.0.10
• 21:13 hasharAway: tagged quibble 0.0.10 a4e88ae
• 20:13 bearND: deployed mobileapps to BC: Config: Start up to 4 workers in parallel during start-up
• 18:55 awight: ORES: Revscoring 2.2.2
• 10:08 _joe_: deleted deployment-mediawiki0[45] and deployment-jobrunner02
• 00:14 legoktm: deployed https://gerrit.wikimedia.org/r/415769

2018-04-24

• 23:49 legoktm: deployed https://gerrit.wikimedia.org/r/428858 https://gerrit.wikimedia.org/r/428859

2018-04-23

• 23:27 ejegg: cleared out /srv/jenkins-workspace/workspace on jenkins-1003
• 21:20 thcipriani: cleaned up 1.5G of space on deployment-tin:/tmp hopefully fixes beta-scap-eqiad
• 20:29 mdholloway: deployed to BC: [mobileapps/deploy@5650605]: Update mobileapps to b011b2a
• 19:09 ottomata: replacing deployment-kafka0[45] with deployment-kafka-main-[12]
• 07:27 hashar: Upgrading Blue Ocean on CI Jenkins 1.4.2 -> 1.5.0

2018-04-20

• 21:32 eddiegp: deployment-kafka04: full disk, deleted /var/log/syslog.1 (3G) to make interactive sessions work again
• 17:30 thcipriani: rebase integration-puppetmaster01:/var/lib/puppet/git conflicting on https://gerrit.wikimedia.org/r/#/c/348236/
• 08:38 hashar: Cleaned integration-slave-docker-1005 disk (deleting workspace and all docker images)
• 08:27 hashar: Created integration-slave-docker-1016 and integration-slave-docker-1017 (2G RAM / 2 executors )
• 08:26 hashar: replaced integration-slave-docker-1010 and integration-slave-docker-1011 with bigger instances (4GB RAM)
• 07:03 hashar: Replaced integration-slave-docker-1001 to get 4GB of RAM
• 00:46 awight: roll back ORES beta to master
• 00:08 awight: Push ORES git-lfs to look at stuff

2018-04-19

• 21:23 hashar: Pooling in the new integration-slave-docker-1002 and integration-slave-docker-1003
• 21:08 hashar: rebuilding integration-slave-docker-1002 and integration-slave-docker-1003 ci1.medium > m1.medium (+2G RAM)
• 21:01 hashar: Bringing back quibble (0.0.9), this time with COMPOSER_PROCESS_TIMEOUT=600 | T192576
• 19:22 hashar: integration-slave-docker-1004 : wiping all jenkins workspace and all docker images
• 18:00 thcipriani: cleared a bunch of old docker images from integration-slave-docker-1004, freed 4.4GB of space
• 17:43 thcipriani: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/#/c/427717/
• 10:34 Hauskatze: fixStuckGlobalRename.php ran to unblock rename jobs never arriving to jobqueue
• 09:41 hashar: building releng/quibble-stretch:0.0.8-2
• 08:19 eddiegp: eddie@deployment-tin:~$ for wiki in deploymentwiki enwiki enwikinews loginwiki metawiki simplewiki; do mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=$wiki --logwiki=loginwiki 'Samtar' "There'sNoTime"; done T192476

2018-04-18

• 22:53 eddiegp: started nutcracker on deployment-jobrunner03
• 22:53 eddiegp: root@deployment-jobrunner03:/var/run# mkdir nutcracker && chown nutcracker.nutcracker nutcracker ref T178457
• 21:16 Hauskatze: (addition to previous message) Cmd ran on deployment-jobrunner03; zillions of messages complaining nutcracker was down
• 21:15 Hauskatze: Ran sudo puppet agent -tv to restart nutcracker, which was "Notice: /Stage[main]/Nutcracker/Service[nutcracker]/ensure: ensure changed 'stopped' to 'running'"
• 20:32 hashar: reloaded zuul for https://gerrit.wikimedia.org/r/427534
• 20:29 Reedy: reloaded zuul for 427469
• 20:11 mdholloway: deployed to BC: [mobileapps/deploy@9328a7d]: Update mobileapps to fb161d7
• 18:03 hauskatze: maurelio@deployment-tin:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=wikidatawiki --logwiki=loginwiki 'Artix Kreiger 2' 'Artix Krieger 2' - T192471
• 17:34 legoktm: deployed https://gerrit.wikimedia.org/r/427421 https://gerrit.wikimedia.org/r/427422 https://gerrit.wikimedia.org/r/427423 a while back
• 13:23 addshore: added Thiemo Kreuz (WMDE) to deployment-prep
• 13:18 addshore: added WMDE-Fisch to deployment-prep
• 09:41 hashar: Triggered https://integration.wikimedia.org/ci/job/quibble-integration/ for REL1_31
• 09:40 hashar: Buiding quibble jobs with (master, mysql, php5.5) https://integration.wikimedia.org/ci/job/quibble-integration/7/ T192432
• 07:17 legoktm: deploying https://gerrit.wikimedia.org/r/426115
• 04:52 legoktm: live hacked /etc/nodepool/wikimedia, trying some things to get rid of mod_php5
• 04:26 legoktm: try once more to refresh nodepool snapshot
• 03:22 legoktm: manually refreshing nodepool ci-jessie image for https://gerrit.wikimedia.org/r/427309

2018-04-17

• 23:40 Krinkle: Rebuilding Nodepool snapshot for https://gerrit.wikimedia.org/r/415312
• 21:10 hasharAway: Running https://integration.wikimedia.org/ci/job/quibble-integration/6/ overnight
• 20:28 hasharAway: building releng/quibble.*:0.0.8 images
• 18:16 Krinkle: Deleted performance-webpagetest-wmf job (removed from config.git)
• 14:32 hashar: Added integration-slave-docker-1008 and integration-slave-docker-1009 (m4executor label)
• 13:33 moritzm: removed role::mediawiki::imagescaler from deployment-mediawiki05, per watroles the only use of that role in WMCS
• 13:32 moritzm: removed role::mediawiki::imagescaler from deployment-prep, per watroles the only use of that role in WMCS
• 02:41 Krinkle: Fix Jenkins config for console section "Composer" end pattern (rm mandatory .+ match from end)

2018-04-16

• 23:13 legoktm: removing obsolete coverage report legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf mediawiki-core-php7
• 22:32 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/427027
• 22:25 legoktm: kick stuch beta-update jenkins jobs
• 20:35 mdholloway: restarted restbase and mobileapps services for testing (T192287)
• 18:58 awight: Update ORES editquality; T185903
• 13:13 _joe_: also shutting down deployment-jobrunner02 T192071
• 13:12 _joe_: manually fixing ssh hostkey for mediawiki-jobrunner03 in scap on deployment-prep
• 12:48 _joe_: turning off deployment-mediawiki{04,05} T192071, will be deleted by EOW
• 02:51 legoktm: deployed https://gerrit.wikimedia.org/r/426837

2018-04-13

• 17:23 marxarelli: Enabling Pipeline Utility Steps plugin in jenkins for changes to service pipeline in https://gerrit.wikimedia.org/r/#/c/425936/
• 16:55 eddiegp: cherry-pick https://gerrit.wikimedia.org/r/#/c/426104/ (test 3)
• 10:02 eddiegp: root@deployment-tin:/mnt/home/jenkins-deploy# sudo -u jenkins-deploy -- sh -c 'ssh-keyscan "deployment-mediawiki-09.deployment-prep.eqiad.wmflabs" >> .ssh/known_hosts' to fix beta-scap-eqiad
• 08:48 eddiegp: debugging T189493 on beta
• 08:20 _joe_: creating deployment-mediawiki-09 with stretch, eliminating -08 which was left in an unusable state T192071
• 07:22 legoktm: restarting jenkins
• 07:21 legoktm: uploading xunit 1.103-wmf.1 to jenkins
• 07:01 hashar: rebuilding quibble containers to use 0.0.7
• 00:53 awight: ORES: Test large file in LFS

2018-04-12

• 23:01 awight: Try gerrit-based submodules for ORES, T180627
• 17:25 thcipriani: running scap pull on deployment-mediawiki07 to catch up from missed beta-scap-eqiad deploys
• 17:23 thcipriani: add ssh host key for deployment-mediawiki07 to /mnt/home/jenkins-deploy/.ssh/known_hosts so that beta-scap-eqiad will work again
• 15:57 thcipriani: integration-slave-docker-1003 had to reinstall python-pbr to fix puppet complaining about tzdata update
• 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch of deployment-prep (T192071)
• 14:24 _joe_: installing deployment-mediawiki{08,09} for the beta upgrade to stretch
• 12:51 hashar: building releng/quibble-jessie and releng/quibble-jessie-php55
• 11:52 _joe_: creating deployment-mediawiki-07, first stretch appserver T192071
• 00:14 twentyafterfour: preparing to deploy phabricator rPHDEP/release/2018-04-12/1 https://phabricator.wikimedia.org/project/view/3335/

2018-04-11

• 21:04 awight: ORES experiment with git-lfs, T180627
• 16:34 Krinkle: `fab deploy_zuul` for https://gerrit.wikimedia.org/r/425551
• 00:03 legoktm: deploying https://gerrit.wikimedia.org/r/425446

2018-04-10

• 13:41 moritzm: upgraded HHVM on mediawiki-deployment04/05/06 to a build with a patch for the MEMC_VAL_COMPRESSION_ZLIB flag in the memcached module (T184854)
• 12:38 hashar: gerrit: created repo operations/debs/tidy-0.99 , a for of tidy Jessie package | T191771

2018-04-09

• 19:50 awight: Redundant virtualenv for ORES
• 18:06 awight: Restore to ORES master branch
• 17:17 awight: Test git-lfs in ORES
• 11:46 Hauskatze: maurelio@deployment-mira:~$ sudo puppet agent -tv to fix T191786 (success: Notice: Applied catalog in 27.11 seconds)

2018-04-06

• 21:51 eddiegp: beta: Cherry-picking https://gerrit.wikimedia.org/r/c/424707/ , test for T173887
• 11:41 moritzm: upgrading deployment-prep to wikidiff2 1.6.0 (T190717)
• 07:40 moritzm: removed mediawiki-deployment07 from deployment-prep (T191578)

2018-04-05

• 21:34 eddiegp: updated deployment-prep cherry-pick of https://gerrit.wikimedia.org/r/c/392221/ to PS38
• 18:08 eddiegp: ran 'cat portals/urls-to-purge.txt | mwscript purgeList.php' on deployment-tin:/srv/mediawiki-staging to clear portals cache
• 17:50 eddiegp: Cherry-picking https://gerrit.wikimedia.org/r/c/424361/ on deployment-puppetmaster02 to try unbreak T173887
• 17:08 bearND: Update mobileapps to dbc0687
• 17:05 hashar: docker: building releng/quibble:0.0.5
• 17:00 bearND: Update mobileapps to 2d5ab5b on beta
• 11:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/424271
• 07:52 moritzm: removed unused/defunct deployment-videoscaler01 from deployment-prep (T191293)
• 07:52 moritzm: removed unused/defunct deployment-tmh01 from deployment-prep (T191293)
• 07:47 hashar: deployment-prep: made Muehlenhoff an admin

2018-04-04

• 22:17 thcipriani: deployment-mediawiki0{4,5} clear apt-cache, restart clear hhvm cache, restart hhvm
• 21:24 awight: Roll back beta ORES
• 20:19 mdholloway: deployed to BC: [mobileapps/deploy@0460519]: Update mobileapps to 2d5ab5b
• 20:12 awight: Try dsh scap config for ORES
• 20:07 addshore: reload zuul for https://gerrit.wikimedia.org/r/424045 (and some quibble experimental stuff)
• 16:09 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/423958
• 15:30 addshore: reload zuul for https://gerrit.wikimedia.org/r/423652
• 12:17 hashar: added experimental quibble job to mediawiki core / vendor / skins/Vector
• 11:58 hashar: Building releng/quibble-stretch:0.0.4
• 11:58 hashar: Building releng/quibble:0.0.4
• 07:11 hashar: deployment-prep: adding EddieGP as a member

2018-04-03

• 09:49 hashar: building releng/quibble:0.0.3
• 00:09 paladox: created new repo "All-Avatars" which will be used to host avatars used by gerrit. Setting owner as Gerrit Managers will allow merging in the repo fro all users soon :)

2018-04-02

• 20:19 mdholloway: deployed to BC: [mobileapps/deploy@940bd48]: Update mobileapps to 58a0a88

2018-03-31

• 21:42 Hauskatze: Ran sudo puppet agent --enable and sudo puppet agent -tv on deployment-maps03 to fix puppet staleness

2018-03-30

• 11:38 dcausse: deployment-prep reindexing with forceSearchIndex all beta wikis (T189694)
• 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001). Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException | T191077
• 09:56 hashar: Nuking /srv/zuul/git/labs/tools/stewardbots on zuul-merger hosts (contint1001 and contint2001). Fetch fails with org.eclipse.jgit.transport.UploadPackInternalServerErrorException | TT191077

2018-03-29

• 19:39 Amir1: ladsgroup@deployment-tin:~$ mwscript updateCollation.php --wiki=fawiki (T190965)
• 19:33 hashar: docker build image releng/quibble-stretch:0.0.2 https://gerrit.wikimedia.org/r/#/c/423025/
• 18:08 greg-g: ran scap task-series blockers --count 26 --date 2018-04-23 1.32.0-wmf.1
• 16:44 hashar: rebuilding docker images for releng/quibble:0.0.1 https://gerrit.wikimedia.org/r/#/c/422959/
• 16:38 hashar: Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/422482/
• 16:35 greg-g: brought compiler02.puppet3-diffs.eqiad.wmflabs back online (herron asked)
• 15:13 paladox: creating new repo mediawiki/extensions/SecureAuth per https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests
• 14:55 mutante: removing deploy1001 from scap masters, keeping it as scap host. to avoid issues syncing (slower) to 3 masters but keep it synced as a regular host (T191029)
• 14:26 elukey: removed from deployment-puppetmaster02's ops repo "1ffd6e72a9 WIP Bug T184234" due to git sync failures

2018-03-28

• 23:42 legoktm: deployed https://gerrit.wikimedia.org/r/422590
• 23:00 legoktm: deployed https://gerrit.wikimedia.org/r/422566 https://gerrit.wikimedia.org/r/422418
• 20:45 bearND: Update mobileapps to a5833a0 on BC
• 19:13 legoktm: killed stuck docker container on 1003 to free up root partition, and then deleted old/all images to free up the rest of the space
• 19:07 legoktm: legoktm@integration-slave-docker-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # full disk
• 16:04 hasharAway: nodepool: deleting 4 instances that are no more used but that Nodepool failed to detect as no omre used (due to some reboots in the openstack infra)

2018-03-27

• 15:23 hashar: building docker-registry.wikimedia.org/releng/operations-puppet:0.3.1 | https://gerrit.wikimedia.org/r/#/c/422168/
• 08:30 kart_: Update cxserver to 9e8ebda
• 03:10 legoktm: deleted mediawiki-core-code-coverage* workspaces to work around git/gerrit issue and retriggered jobs

2018-03-26

• 20:43 mdholloway: deployed to BC: [mobileapps/deploy@e223f51]: Update mobileapps to 534f95d
• 13:55 hashar: restarting CI Jenkins . Upgrades Mail plugin from 1.20 to 1.21 | T190393
• 03:35 bd808: Marked integration-slave-docker-1003 offline because / is full. Needs cleanup. (cc hashar, thcipriani)

2018-03-23

• 21:19 greg-g: gjg@deployment-tin:/srv/mediawiki-staging/php-master$ git remote prune origin

2018-03-22

• 12:28 elukey: removed two patches from deployment-puppetmaster02's operations/puppet repo because git-sync was failing (https://gerrit.wikimedia.org/r/#/c/361648/ and https://gerrit.wikimedia.org/r/#/c/402758/)

2018-03-21

• 23:25 RoanKattouw: Created maps security group for port 6533; removed port 6533 from sca security group
• 23:22 bd808: Raised security group quota from 20 to 40
• 22:40 hashar: cancel beta-code-update build in Jenkins (deadlock)
• 22:40 hashar: cancel beta-update-database-eqiad build in Jenkins (deadlock)
• 20:19 mdholloway: deployed to BC: [mobileapps/deploy@675837f]: Update mobileapps to e6b50a0
• 18:40 bd808: Re-added ArielGlenn as deployment-prep project admin
• 18:18 elukey: fixing operations/puppet on deployment-puppetmaster02, git-sync stuck due to a old patch still there
• 17:52 hashar: deployment-prep: scap sync-file docroot/wwwportal/portal 'T173887'
• 00:43 Reedy: enabled compiler03.puppet3-diffs.eqiad.wmflabs and disabled compiler02.puppet3-diffs.eqiad.wmflabs in jenkins

2018-03-20

• 19:04 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420777
• 17:27 mdholloway: deployed to BC: [mobileapps/deploy@fad1009]: Update mobileapps to 634a15f
• 12:25 hashar: deployment-prep: added Muehlenhoff (moritzm) as a member
• 10:35 elukey: stop eventlogging + mysql on deployment-eventlog05 for maintenance
• 09:58 hashar: deployment-mira: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files
• 09:38 hashar: deployment-tin: deleted iegreview/scholarships files scap/checks.yaml (invalid yaml somehow), ran puppet and restore the files

2018-03-19

• 18:34 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420398/
• 16:57 awight: ORES beta service is restored.
• 16:45 awight: Put ORES-beta back onto master branch
• 16:43 awight: ORES-beta has been down since January.
• 16:29 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/420373/
• 16:11 bd808: jenkins: compiler03.puppet3-diffs offline for https://phabricator.wikimedia.org/T187258
• 15:20 thcipriani: reinit /srv/mediawiki-staging/portals on deployment-tin to fix beta-mediawiki-config-update-eqiad
• 11:56 hashar: Creating docker container docker-registry.wikimedia.org/releng/java8-xgboost:0.1.1 | https://gerrit.wikimedia.org/r/#/c/420311/ | T190032
• 10:50 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/419479/
• 07:43 kart_: Add cx_version to cx_translations table (T190009)

2018-03-18

• 22:22 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/420230
• 04:42 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/417344

2018-03-16

• 12:56 hashar: deployment-tin: ran git remote prune origin / git gc on all /srv/deployment git repositories
• 12:56 hashar: deployment-tin: setting /srv/deployment files/dir to be group owned by wikidev and group writable
• 12:50 hashar: deployment-tin: deleting /srv/grafana (no more in Gerrit)
• 12:46 hashar: deployment-tin: sudo chown mwdeploy:mwdeploy /srv/mediawiki/.git/objects/pack/* # some pack of 6GB belonged to root
• 11:11 hashar: zuul: reenqueue all coverage jobs lost when restarting Zuul
• 10:53 hashar: Upgrading zuul to zuul_2.5.1-wmf4 to resolve a mutex deadlock T189859
• 02:44 thcipriani: beta-scap-eqiad stuck 14 hours, had to do disconnect reconnect dance

2018-03-15

• 17:15 bearND: Update mobileapps to c5e1522
• 16:43 Amir1: created deployment-ores01 to use stretch
• 16:36 Amir1: deleting deployment-ores01 to reimage
• 15:47 awight: ORES with git-lfs, scap config
• 09:24 hashar: deployment-prep: added gehel and dcausse has members/admins of the labs project
• 00:21 awight: ORES with git-lfs

2018-03-14

• 23:30 awight: new ORES submodule, pre-git-lfs
• 20:23 mdholloway: deployed to BC: [mobileapps/deploy@0f9625a]: Update mobileapps to 9f4a80c
• 19:43 no_justification: deployment-prep: fixed permissions on i18n cache to unbreak scap job
• 16:36 Hauskatze: maurelio@deployment-tin:~$ sudo chown -R jenkins-deploy:wikidev /srv/mediawiki-staging/php-master
• 15:44 addshore: killed https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3382/ and https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/143/ to unblock integration slaves
• 14:23 Reedy: beta-scap-eqiad disabled while freeing up space on deployment-tin
• 12:16 kart_: update cxserver to c355d0c
• 10:07 hashar: nodepool: Image snapshot-ci-jessie-1521021138 in wmflabs-eqiad is ready
• 09:52 hashar: Refreshing Nodepool snapshot for "mw-apply-settings: get LocalSettings.php archived" https://gerrit.wikimedia.org/r/#/c/419375/
• 08:05 hashar: Move Jenkins jobs mwext-phpunit-coverage-patch mwext-phpunit-coverage-publish mediawiki-phpunit-coverage-patch to Nodepool (they use too much disk space) | T189365

2018-03-13

• 21:47 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
• 18:08 awight: Enable Extension:JADE, T176333
• 11:26 kart_: Update cxserver to bd2ccfc
• 10:39 kart_: Update cxserver to fd2c4be
• 09:50 hashar: integration: cherry pick https://gerrit.wikimedia.org/r/#/c/419131/1 | T188112
• 09:46 hashar: integration: cherry pick PS3 of https://gerrit.wikimedia.org/r/#/c/412894/ | T188112
• 08:13 hashar: integration-cumin: ln -s /usr/local/lib/python3.4 /usr/local/lib/python3 (HACK) | T188112
• 08:09 hashar: integration: cherry picking https://gerrit.wikimedia.org/r/c/412894/ to fix cumin 3.0.1 | T188112

2018-03-12

• 20:13 mdholloway: deployed to BC: [mobileapps/deploy@c764714]: Update mobileapps to 5c90db7
• 20:09 bd808: Forced puppet run on deployment-logstash2 for {{gerrit|418986}}
• 20:09 bd808: Removed role::logstash::eventlogging from deployment-logstash2 because the hiera config is failing (undefined method `[]' for nil:NilClass at /etc/puppet/modules/role/manifests/logstash/eventlogging.pp:11)
• 19:55 bd808: Forced puppet run on deployment-logstash2. Failed due to bad logstash::eventlogging hiera data
• 19:54 bd808: Cherry picked https://gerrit.wikimedia.org/r/#/c/418986/ to deployment-puppetmaster02
• 12:33 hashar: Jenkins: installed Blue Ocean plugin. Eg: https://integration.wikimedia.org/ci/blue/ | T155840
• 12:24 hashar: Jenkins: uninstalled the "cvs" plugin

2018-03-10

• 07:03 greg-g: gjg@integration-slave-jessie-1004:/srv/jenkins-workspace/workspace$ sudo rm -rf * - T189365

2018-03-09

• 22:01 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *
• 20:10 legoktm: deployed https://gerrit.wikimedia.org/r/418024
• 16:06 hashar: Deploying docker releng/npm-test:0.5.0 https://gerrit.wikimedia.org/r/#/c/417960/
• 10:58 hashar: Polling 6 new Docker instances to jenkins: integration-slave-docker 1010 to 1015. They are ci.medium (2G RAM / 2vcpu) each with 2 executors and labels DebianJessieDocker, m1executor
• 10:37 hashar: Update *-maven-java8-docker-site-publish jobs which were not mounting /src into the container | T188686
• 02:31 legoktm: deployed https://gerrit.wikimedia.org/r/417343

2018-03-08

• 22:19 hasharDinner: cleaned up /srv on integration-slave-jessie-1001 . Upgrade packages and reboot.
• 21:59 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf * # out of disk space
• 18:36 bearND: Update mobileapps to afb0167
• 17:13 hashar: deleting a few nodepool instances that are no more registered in Jenkins
• 14:12 hashar: deployment-tin: rm -fR /srv/ocg
• 14:03 hashar: deployment-tin: rm /srv/jenkins/home/jenkins-deploy/workspace/beta-scap-eqiad/central.hhbc # 1.4GBytes
• 14:02 hashar: deployment-tin is out of disk space on /srv
• 10:27 hashar: Deploy docker images for /deploy repositories | https://gerrit.wikimedia.org/r/#/c/417217/

2018-03-07

• 11:08 hashar: reloading Zuul for "Add search/analysis-extra to jenkins" | T188686
• 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/416886 https://gerrit.wikimedia.org/r/415588
• 02:28 legoktm: manually triggering jenkins jobs

2018-03-06

• 20:59 hashar: gerrit: changed scoring/ores/assets parent permission group to scoring/ores
• 20:59 hashar: gerrit: created scoring/ores/draftquality scoring/ores/drafttopic scoring/ores/articlequality scoring/ores/editquality scoring/ores/deploy all inherit permissions from scoring/ores itself inheriting from scoring/
• 20:56 hashar: gerrit: created scoring/ parent project with owner being research-ores ( https://gerrit.wikimedia.org/r/#/admin/projects/scoring,access )
• 20:52 MaxSem: refreshing spoofuser on beta
• 19:34 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/TorBlock/maintenance/loadExitNodes.php --force
• 19:31 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
• 18:34 mdholloway: deployed to beta: [mobileapps/deploy@5986ab7]: Update mobileapps to afbe9af
• 03:43 Krinkle: Jenkins postmerge queue has 'beta-scap-eqiad' and 'beta-update-databases-eqiad' stuck "Waitinf for execute" for over 3h

2018-03-05

• 16:38 Reedy: deleted the stack traces too
• 16:37 Reedy: removed pre 2018 hhvm error logs from deployment-mediawiki04
• 16:37 Reedy: that was from deployment-mediawiki04
• 16:35 Reedy: removed 2G temp folder from /srv/mediawiki/php-master/cache/l10n/upstream
• 06:14 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2018-03-04

• 19:00 thcipriani: cleared /tmp, apt-cache deployment-mediawiki04
• 06:02 Krinkle: Re-create php-master/StartProfiler.php on deployment-tin in Beta Cluster, similar to the one scap auto-creates each week in prod, except to include StartProfiler-labs.php instead.
• 02:12 Krenair: Regenerated captcha images for T164047

2018-03-02

• 20:49 legoktm: deployed https://gerrit.wikimedia.org/r/411185
• 20:07 legoktm: deployed https://gerrit.wikimedia.org/r/415925 https://gerrit.wikimedia.org/r/414069
• 10:21 legoktm: deployed https://gerrit.wikimedia.org/r/415823 https://gerrit.wikimedia.org/r/415824 https://gerrit.wikimedia.org/r/415825
• 10:16 legoktm: deployed https://gerrit.wikimedia.org/r/415819
• 09:09 legoktm: deployed https://gerrit.wikimedia.org/r/415553
• 08:47 legoktm: deploying https://gerrit.wikimedia.org/r/415809
• 08:09 legoktm: manually queing jenkins jobs

2018-03-01

• 20:48 Hauskatze: maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php
• 14:55 elukey: delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch)
• 02:11 legoktm: manually queued jenkins jobs

2018-02-28

• 16:31 legoktm: manually queued jenkins jobs
• 10:51 hashar: integration-slave-jessie-android killed stall qemu-system-i386 process
• 10:42 hashar: build docker-registry.discovery.wmnet/releng/npm-browser-test:0.1.2 and docker-registry.discovery.wmnet/releng/npm-test-oojsui:0.1.1
• 08:42 legoktm: queued more jenkins jobs (last for tonight)
• 07:29 legoktm: mass queuing jenkins jobs again

2018-02-27

• 21:17 hashar: Building docker image releng/npm-test-oojsui:0.1.0 - https://gerrit.wikimedia.org/r/#/c/415102/
• 10:15 zeljkof: Reloading Zuul to deploy d9ed9d4
• 09:35 hashar: deployment-mediawiki05: out of disk space. Ran apt-get clean, cleaned old kernels/packages and dropped hhvm bytecode cache
• 08:44 hashar: deployment-mediawiki06: out of disk space. Ran apt-get clean
• 07:40 legoktm: deployed https://gerrit.wikimedia.org/r/414957
• 06:11 legoktm: manually triggering a bunch of jenkins jobs
• 03:29 legoktm: deployed https://gerrit.wikimedia.org/r/414896
• 03:02 Krinkle: Deleted beta-* related job builds in Jenkins that were stuck >1hr
• 03:01 Krinkle: Jenkins slave connection to deployment-tin is broken again. No error. Script console works. Disconnect/Relaunch doesn't resolve. 6 idle executors but jobs are no starting for some reason.

2018-02-26

• 22:40 Hauskatze: updating list of Tor nodes for TorBlock on Beta Cluster wikis
• 22:39 Hauskatze: purging old abusefilter IP data from Beta Cluster wikis while we wait for a cron job to do this automatically
• 18:39 mutante: deployment-cache-text04 - manually creating Letsencrypt SSL cert for fr.wikipedia.beta.wmflabs.org (acme-setup -i "fr_wikipedia_beta_wmflabs_org" -s "fr.wikipedia.beta.wmflabs.org" --key-user root --key-group root), restarted nginx (T188288)

2018-02-24

• 02:21 legoktm: deployed https://gerrit.wikimedia.org/r/413964

2018-02-23

• 16:52 elukey: created deployment-eventlogging05 to test eventlogging on Debian in deployment-prep
• 11:50 hashar: deployment-mediawiki04 : apt-get clean && apt-get autoremove --purge
• 11:50 hashar: deployment-mediawiki04 : rm /var/cache/hhvm/*.sq3 and restarting hhvm
• 11:48 hashar: deployment-mediawiki04 is out of disk space on / causing beta-scap-eqiad to fail
• 08:19 hashar: gerrit: marked apps/android/java-mwapi.git read-only | T187995

2018-02-22

• 14:58 hashar: build docker-registry.discovery.wmnet/releng/tox-mysqld:0.1.0 | https://gerrit.wikimedia.org/r/#/c/413374/
• 10:23 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.2 for https://gerrit.wikimedia.org/r/#/c/413336/
• 09:43 hashar: Building docker-registry.discovery.wmnet/releng/npm-test-3d2png:0.1.1 for https://gerrit.wikimedia.org/r/#/c/413286/
• 00:27 mdholloway: deployed to BC [mobileapps/deploy@8ffb03b]: Update mobileapps to a1339a9

2018-02-21

• 22:31 hashar: Building docker image releng/npm-test-3d2png:0.1.0 and reloading Zuul | https://gerrit.wikimedia.org/r/413227
• 20:51 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/413060
• 16:15 zeljkof: Reloading Zuul to deploy d90d617
• 11:26 Hauskatze: Refreshed TOR exit nodes & cleaned old abusefilter log data for all Beta Cluster wikis.
• 02:39 no_justification: beta: ran initSiteStats.php --update on all wikis

2018-02-20

• 20:57 Hauskatze: Refreshed tor exit node lists for all Beta Cluster wikis
• 20:50 legoktm: refreshing nodepool image: nodepool@labnodepool1001:~$ nodepool image-update wmflabs-eqiad snapshot-ci-jessie
• 20:47 HausAFKatze: mwscript extensions/TorBlock/maintenance/loadExitNodes.php --wiki=deploymentwiki --force
• 19:41 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php

2018-02-19

• 23:40 Krinkle: Running `nodepool image-update wmflabs-eqiad snapshot-ci-jessie` to deploy https://gerrit.wikimedia.org/r/412825
• 18:52 legoktm: deleted all current nodepool instances for ci-jessie
• 18:36 legoktm: manually refreshing nodepool images (nodepool image-update wmflabs-eqiad snapshot-ci-jessie)
• 10:03 hashar_: deployment-tin: git gc in /srv/mediawiki-staging/php-master and /srv/mediawiki-staging/php-master/extensions

2018-02-18

• 20:37 Hauskatze: Ran foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php on Beta

2018-02-16

• 21:26 legoktm: deploying https://gerrit.wikimedia.org/r/411186
• 21:12 hashar: Upgraded Zuul to https://gerrit.wikimedia.org/r/#/c/411322/3
• 15:39 hashar: castor02: sudo rm -fR /srv/jenkins-workspace/caches/*/*/*trusty
• 13:56 hashar: Jenkins: disabled JNLP protocol entirely | https://integration.wikimedia.org/ci/configureSecurity/
• 11:04 hashar: updating all docker related jobs https://gerrit.wikimedia.org/r/#/c/411098/
• 07:21 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2018-02-15

• 23:30 Hauskatze: Ran cleanupSpam.php on deploymentwiki to get rid of a bunch of crap.
• 20:34 hashar: Updating docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/#/c/410990/ (no jobs touched)
• 10:35 hashar: gerrit: marking search/ltr and search/repository-swift as read-only | T187428
• 07:26 legoktm: cancelled a job of mediawiki-core-doxygen-publish to unstuck them
• 07:13 legoktm: deployed https://gerrit.wikimedia.org/r/410654
• 01:41 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/410255

2018-02-14

• 21:41 mdholloway: Deployed to beta cluster: [mobileapps/deploy@9bad612]: Update mobileapps to f23519f
• 20:23 legoktm: deploying https://gerrit.wikimedia.org/r/410554
• 20:21 hashar: building docker-registry.wikimedia.org/releng/tox-pyspark:0.1.1
• 17:37 hashar: Manually deleting some Nodepool instances following Jenkins restart
• 17:37 hashar: Upgraded Jenkins to 2.89.4
• 13:31 Hauskatze: maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldLogIPData.php > purge_abusefilter_20180214.log 2>&1
• 09:41 hashar: restart Docker on all Jenkins agents
• 08:26 legoktm: deployed https://gerrit.wikimedia.org/r/410397
• 04:28 legoktm: CodeEditor and WikiEditor are now part of shared MediaWiki gate job
• 04:03 legoktm: deployed https://gerrit.wikimedia.org/r/410254
• 03:50 legoktm: deployed https://gerrit.wikimedia.org/r/410194
• 03:33 legoktm: deploying https://gerrit.wikimedia.org/r/410372
• 03:12 legoktm: deploying https://gerrit.wikimedia.org/r/410005
• 02:23 MaxSem: manually created the global_preferences table on betalabs' centralauth DB

2018-02-13

• 18:19 mdholloway: deployed mobileapps@5851dfc to beta cluster
• 16:00 legoktm: deployed https://gerrit.wikimedia.org/r/410173 https://gerrit.wikimedia.org/r/410192
• 10:51 moritzm: shutdown deployment-videoscaler01 (T187063)
• 10:49 moritzm: shutdown deployment-tmh01 (T187063)
• 10:21 Hauskatze: purged old IP data from abuse_filter_log
• 09:15 hashar: beta: scap fixed by deleting a leftover /srv/mediawiki/.~tmp~/
• 09:12 hashar: deployment-jobrunner02: sudo rm -fR /srv/mediawiki/.~tmp~/ # might be causing the beta-scap-eqiad Jenkins job to fail
• 03:44 legoktm: deployed https://gerrit.wikimedia.org/r/410100
• 01:52 legoktm: deploying https://gerrit.wikimedia.org/r/410089

2018-02-12

• 23:06 legoktm: deployed https://gerrit.wikimedia.org/r/410075 https://gerrit.wikimedia.org/r/410076
• 20:53 mdholloway: deployed mobileapps@f14bdd5 to beta cluster
• 18:55 Hauskatze: Running maurelio@deployment-tin:~$ foreachwikiindblist all-labs.dblist extensions/AbuseFilter/maintenance/purgeOldIPLogData.php for T186870
• 18:48 Hauskatze: Restoring first missing log entries on Beta refs. T54919
• 18:46 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=arwiki (37 rows purged - T186870)
• 18:45 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged - T186870)
• 18:44 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=aawiki (0 rows purged)
• 18:43 Hauskatze: Starting to purge old afl_ip data from abuse_filter_log on Beta Cluster - T186870
• 15:59 hashar: Deploying java8 docker image https://gerrit.wikimedia.org/r/#/c/409881/

2018-02-10

• 18:58 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/purgeOldLogIPData.php --wiki=eswiki (1695 rows purged - T186870)
• 18:49 Hauskatze: maurelio@deployment-tin:~$ mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=zhwiki (22 rows missing inserted)
• 18:46 Hauskatze: Ran mwscript extensions/AbuseFilter/maintenance/addMissingLoggingEntries.php --wiki=deploymentwiki (17 rows inserted)

2018-02-09

• 23:05 legoktm: deploying https://gerrit.wikimedia.org/r/409461
• 22:59 legoktm: deployed https://gerrit.wikimedia.org/r/409459
• 22:40 legoktm: deploying https://gerrit.wikimedia.org/r/409458
• 21:53 legoktm: deploying https://gerrit.wikimedia.org/r/408760
• 21:16 legoktm: deleted test-csteipp-sensiolabs-securityadvisorieschecker from jenkins, obsolete
• 13:43 hashar: Deploying docker-registry.wikimedia.org/releng/jsduck:0.1.0 | https://gerrit.wikimedia.org/r/#/c/409320/
• 10:16 hashar: Launching new integration-slave-docker-1003 T186856
• 09:48 hashar: Deleting integration-slave-docker-1003 and recreating it. It is running on an overloaded labvirt machine. | T179378 T186856
• 09:25 legoktm: deploying https://gerrit.wikimedia.org/r/409297
• 08:53 hashar: Reloading zuul for I73a314
• 07:13 legoktm: marked integration-slave-docker-1003 as offline - T186856
• 00:59 bd808: Removed Yuvipanda at user request (T186289)
• 00:58 bd808: Removed Yuvipanda at user request (T186289)

2018-02-08

• 13:58 Hauskatze: maurelio@deployment-tin:~$ mwscript initSiteStats.php --wiki=deploymentwiki --update --active --use-master
• 10:02 hashar: Rebuilding docker-pkg images on contint1001. Would get chromium 64 into npm-browser-test | T179552

2018-02-07

• 21:56 legoktm: deploying https://gerrit.wikimedia.org/r/408781
• 03:15 legoktm: deployed https://gerrit.wikimedia.org/r/408746
• 02:57 legoktm: deploying https://gerrit.wikimedia.org/r/408368
• 01:19 legoktm: deploying https://gerrit.wikimedia.org/r/408735
• 01:00 legoktm: deploying https://gerrit.wikimedia.org/r/408230

2018-02-06

• 21:41 hashar: Rebuilding Zuul package to hotfix T186381
• 21:14 legoktm: restarted zuul due to patch being stuck (T186381)
• 19:25 hashar: Restarted Zuul due to T186381
• 18:14 thcipriani: removing /srv/mediawiki/.git on deployment-tin to clear space
• 02:33 legoktm: deploying https://gerrit.wikimedia.org/r/408480

2018-02-05

• 22:09 mdholloway: deployed mobileapps@3140b1a to BC
• 16:09 mdholloway: mobileapps deployment to BC failed with error (T186532)
• 12:37 Hauskatze: deployment-prep maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki *.doxawatches.com --delete
• 11:02 hashar: Upgrading jenkins-debian-glue to 0.18.4-wmf1 | T186494
• 09:48 hashar: operations/debs/jenkins-debian-glue create branches debian/jessie-wikimedia and patch-queue/debian/jessie-wikimedia based on v0.17.0 | T186494

2018-02-04

• 23:47 legoktm[NE]: deploying/building https://gerrit.wikimedia.org/r/408228

2018-02-03

• 21:05 legoktm: manually deleted /srv/zuul/git/mediawiki/tools/phan on contint1001 so zuul could clone the new repo
• 21:02 legoktm: deployed https://gerrit.wikimedia.org/r/407991
• 04:01 legoktm: disable/enabled gearman in jenkins
• 03:55 legoktm: restarting zuul to drop 407165,3 from the queue
• 03:48 legoktm: disabled/enabled gearman in jenkins

2018-02-01

• 16:14 Amir1: deleting deployment-sca03 (T184501)
• 07:09 legoktm: legoktm@integration-slave-jessie-1001:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2018-01-31

• 22:01 mdholloway: updated mobileapps to 3d717fa on beta cluster
• 05:29 legoktm: brought integration-slave-jessie-1003 back online after clearing disk space
• 05:28 legoktm: legoktm@integration-slave-jessie-1003:/srv/jenkins-workspace/workspace$ sudo rm -rf *

2018-01-30

• 20:09 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/404778
• 13:23 Amir1: made User:Ladsgroup admin and 'crat in wikidatawiki

2018-01-29

• 23:24 awight: Experiment with versioned ORES venv, T181071

2018-01-24

• 23:14 Krenair: armed keyholder on deployment-cumin using deployment-puppetmaster02:/var/lib/git/labs/private/files/ssh/tin/cumin_rsa.passphrase - this seems to have fixed cumin

2018-01-23

• 20:50 legoktm: deployed https://gerrit.wikimedia.org/r/405941
• 19:56 Krinkle: Reloading Zuul to deploy https://gerrit.wikimedia.org/r/405930
• 18:25 Hauskatze: Ran fixDoubleRedirects.php on deploymentwiki
• 18:20 Hauskatze: maurelio@deployment-tin:~$ mwscript cleanupSpam.php --wiki=deploymentwiki --delete *.bestrealmoneygames.com
• 15:58 hashar: integration: armed keyholder on integration-cumin using passphrase from integration-puppetmaster01.integration.eqiad.wmflabs:/var/lib/git/labs/private/files/ssh/cumin.passphrase | https://wikitech.wikimedia.org/wiki/Keyholder

2018-01-22

• afk: restarting jenkins

2018-01-20

• 09:09 legoktm: reloading zuul to deploy https://gerrit.wikimedia.org/r/405404 https://gerrit.wikimedia.org/r/405352 https://gerrit.wikimedia.org/r/404972 https://gerrit.wikimedia.org/r/404229 https://gerrit.wikimedia.org/r/404230 https://gerrit.wikimedia.org/r/354667

2018-01-19

• 17:23 zeljkof: Reloading Zuul to deploy 25cab6f
• 17:10 zeljkof: Reloading Zuul to deploy 7d6b4ee
• 15:34 elukey: added deployment-eventlog02.deployment-prep.eqiad.wmflabs to /etc/ssh/ssh_known_hosts on deployment-tin (following https://phabricator.wikimedia.org/T116206#2251441) to unblock "Host key verification failed" for Analytics

2018-01-18

• 18:29 bearND: (beta): Update mobileapps to 2690899
• 17:00 ottomata: stashing local changes to deployment-puppetmaster02 in /var/lib/git/operations/puppet (mail/mx.pp and exim/exim4.conf.mx.erb)
• afk: cleared some space on deployment-mediawiki05 (apt-cache and old logs) so scap had room to work again. Although space is tight.
• 11:49 legoktm: mediawiki-core-doxygen-publish jobs are stuck
• 00:21 bd808: Deleted 6-7 nodepool instances in an alive but offline for running jobs state

2018-01-17

• 16:24 zeljkof: Reloading Zuul to deploy 5f75731
• 13:11 hashar: nodepool: updating snapshot to get hhvm +wmf4 for T185024  : nodepool image-update wmflabs-eqiad snapshot-ci-jessie

2018-01-16

• 21:29 thcipriani: armed keyholder using instructions on https://wikitech.wikimedia.org/wiki/Keyholder
• 20:20 thcipriani: reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/403904/1
• 18:16 greg-g: muted shinken in -releng while the reboots are on-going
• 09:38 hashar: Deleting all legacy wmfreleng/ docker images from the Jenkins slaves : sudo cumin --force 'name:docker' "docker images|grep wmfreleng|awk '{print \$3}'|sort|uniq|xargs docker rmi --force"
• 09:27 hashar: deploy rake/rake-vagrant docker images | https://gerrit.wikimedia.org/r/#/c/404422/

2018-01-15

• 21:59 hashar: deployment-mx echo -n > /var/log/mtail/mtail.log
• 21:59 hashar: deployment-mx rm /var/log/git-sync-upstream.log*
• 17:25 zeljkof: Reloading Zuul to deploy ff0a02d
• 17:14 zeljkof: Reloading Zuul to deploy fff6431
• 13:59 kart_: Ran: "mwscript extensions/Translate/scripts/createMessageIndex.php --wiki=metawiki" for T180841
• 13:37 kart_: Ran update.php on metawiki betacluster (T180841)
• 11:23 hashar: Mirroring git://anonscm.debian.org/pkg-php/php-ast.git to operations/debs/pkg-php/php-ast.git | T174338
• 09:50 hashar: integration/zuul pushed upstream git tags to our repo

2018-01-12

• 14:11 zeljkof: Reloading Zuul to deploy 3816837

2018-01-11

• 17:50 tgr: added Groovier1 to project members for T158909
• 14:05 hashar: Migrate composer-php70-docker mwgate-composer-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403654
• 13:45 hashar: Migrate composer-package-php70-docker mwgate-composer-package-php70-docker to a new docker image https://gerrit.wikimedia.org/r/403647

2018-01-10

• 20:55 hasharAway: Updating '*rake-docker' jobs https://gerrit.wikimedia.org/r/#/c/403471/
• 20:54 hasharAway: Updating rake / rake-vagrant docker containers | https://gerrit.wikimedia.org/r/#/c/403471/
• 20:45 apergos: that should be of course deployment-puppetmaster02
• 20:43 apergos: replaced /var/lib/git/labs/private on deployment-puppetmaster0 with a clean copy of the repo with all uncommitted changes checked in, all extra files checked in, all local commits rebased on top; old repo is in /var/lib/git/labs/private_icky
• 19:38 hasharAway: updating docker-pkg images for rake-vagrant https://gerrit.wikimedia.org/r/#/c/403462/
• 14:48 _joe_: shutting down deployment-puppetdb01.deployment-prep.eqiad.wmflabs, unused
• 09:40 hashar: update docker-pkg images for releng/rake https://gerrit.wikimedia.org/r/#/c/403311/
• 07:50 legoktm: deployed https://gerrit.wikimedia.org/r/402826

2018-01-09

• 14:53 hashar: Change integration/zuul.git HEAD from 'master' to 'patch-queue/debian/jessie-wikimedia' | T158243 T162191
• 13:20 addshore: reloaded zuul to deploy https://gerrit.wikimedia.org/r/402862
• 10:54 hashar: gerrit: created operations/debs/node-tunnel-agent a fork of git://anonscm.debian.org/collab-maint/node-tunnel-agent.git | T183569
• 08:51 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki (T184276)
• 08:49 Amir1: ladsgroup@deployment-sca03:/srv/deployment$ sudo rm -rf ores (T184282)
• 08:48 Amir1: stopping ores services in deployment-sca03 (T184282)
• 08:43 Amir1: changed DNS that ores-beta.wmflabs.org points to deployment-ores01 instead of deployment-sca03
• 08:42 Amir1: deleted deployment-ores-redis-01 in favor of deployment-ores01 (T184282)

2018-01-08

• 18:32 thcipriani: doing https://www.mediawiki.org/wiki/Continuous_integration/Architecture/Troubleshooting#Jenkins_executioner_lock to fix deployment-tin executioner lock stalling postmerge.
• 16:32 addshore: reloaded zuul for https://gerrit.wikimedia.org/r/#/c/383872/

2018-01-05

• 21:02 legoktm: legoktm@contint1001:/srv/org/wikimedia/doc/cover$ sudo -u jenkins-slave rm -rf extensions
• 14:26 halfak: restarted celery-ores-worker on deployment-sca03

2018-01-03

• 19:55 hashar: manually upgrading puppet to 4.8 on deployment-mx / deployment-redis01 / deployment-redis02 | T184114
• 19:52 hashar: purging old kernels on deployment-mx / deployment-redis01 / deployment-redis02 | T184114
• 19:24 hashar: deployment-prep: fix puppet run broken by a duplicate definition due to profile::base::firewall vs base::firewall
• 19:20 hashar: deployment-tin "upgrade" scap to 3.7.4-3 the version in apt.wm.o
• 19:09 hashar: apt-get upgrade on deployment-tin . "downgrade" scap from 3.7.4-3 (apt.wm.o) to 3.7.4-1~20180103034049.266 (from CI)
• 19:07 hashar: deployment-prep: restored all the cherry picks on the puppet master
• 06:01 legoktm: manually installing php-xdebug on integration-slave-jessie-1004 to make sure this works (temporary)

2018-01-01

• 23:46 Krenair: ran `mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=sqwiki` on deployment-tin for T183862
• 19:56 Amir1: ladsgroup@deployment-tin:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php --wiki=nlwiki (T183862)
• 19:56 Amir1: restarting ores services in deployment-sca03 (T183862)

Archives

• Archive 1 (September 2014 - December 2015)
• Archive 2 (2016)
• Archive 3 (2017)
• Archive 4 (2018)
• Archive 5 (2019)
• Archive 6 (2020)
• Archive 7 (2021)
• Archive 8 (2022)
• Archive 9 (2023)