Jump to navigation Jump to search
- 18:16 mutante: bastion-restricted-eqiad1-01 - /etc/apt/sources.list.d# mv openstack-mitaka-jessie.list /root/ ; puppet agent -tv - fixed puppet run T281176
- 18:14 mutante: bastion-restricted-eqiad1-01, a stretch host, has openstack-mitaka-jessie.list in APT sources. mixed stretch/jessie sources cause issues T281176
- 18:08 mutante: bastion-restricted-eqiad1-01 puppet runs again but still has unrelated issues, like "failed to fetch python3-dateutil" T281176 on stretch
- 18:06 mutante: running puppet on bastion-restricted-eqiad1-01 after deploying fix for T281176
- 18:32 bd808: `sudo wmcs-openstack role remove --user zfilipin --project bastion user` T267313
- 14:57 Reedy: killed meno25 process running pwb.py on tools-sgebastion-07
- 00:41 Krenair: T245932
- 12:07 arturo: T223148 move bastion-eqiad1-02 to cloudvirt1001
- 16:10 Krenair: Changed bastion-restricted* hieradata to account for https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/500640/ T220990
- 19:43 andrewbogott: deleting old eqiad-region bastions
- 17:15 andrewbogott: shutting down all bastions in the eqiad region
- 14:32 gtirloni: applied kernel update and rebooted all bastion VMs
- 14:15 gtirloni: applied kernel update and rebooted bastion-restricted-eqiad1-01
- 16:16 arturo: force-rebooted again bastion-restricted-01 due to some additional tests with DHCP
- 11:22 arturo: rebooting bastion-restricted-01 to test new setup from T202636
- 16:58 arturo: add myself as project admin
- 16:13 arturo: force-reboot bastion-restricted-01 bc it lost the IP address, and to force a new DHCP query
- 05:35 bd808: Added Chico Venancio as project member (T185273)
- 05:35 bd808: Added Chico Venancio as project member
- 13:56 chasemp: add profile::openstack::main::cumin_auth_group: cumin_real_masters for puppet prefix bastion-restricted
- 21:33 bd808: Restarted nscd, nslcd on bastion-02.bastion.eqiad.wmflabs
- 21:16 bd808: Forced puppet run and restarted nslcd on bastion-01.bastion.eqiad.wmflabs
- 15:11 bd808: Added self (BryanDavis) as projectadmin
- 00:18 yuvipanda: added Krenair as admin to help with T132225 and other issues
- 23:46 YuviPanda: terminated mwds-proxy instance + floating IP
- 18:49 YuviPanda: deleted bastion1 and bastion2 instances. RIP
- 20:18 Coren: getting rid of old backups-of-backups of ancient filesystems
- 20:17 YuviPanda: deleted bastion3 after alerting sole user in the last few weeks (Krenair)
- 14:05 YuviPanda: removed bastion-restricted-pdns, released IP and hostname
- 14:00 YuviPanda: deleted bastion-restricted1 host, moved bastion-restricted.wmflabs.org to bastion-restricted-01
- 18:16 YuviPanda: created bastion-01, -02, -03 as debian jessie instances.
- 22:30 Coren: rebooted bastion1 and bastion2
- 06:30 andrewbogott: rebooting bastion1
- 08:13 andrewbogott: rebooted bastion2 due to virt11 issues
- 16:17 andrewbogott: rebooting bastion1
- 22:34 hashar: ping
- 22:50 mutante: creating new project 'netflow'
- 10:47 hashar: labstore3 Faidon rebooted labstore3 and restarted NFS. Issue solved.
- 10:46 hashar: labstore3 showed up a system CPU plateau starting at roughly 10:08am ( http://ganglia.wikimedia.org/latest/?r=hour&s=by+name&c=Labs%2520NFS%2520cluster%2520pmtpa ) seems NFS is dead now.
- 10:41 hashar: labstore3 shows up a system CPU plateau ( http://ganglia.wikimedia.org/latest/?r=hour&s=by+name&c=Labs%2520NFS%2520cluster%2520pmtpa ) seems NFS is dead now.
- 20:50 andrewbogott: restarted puppet service on all bastion hosts -- it had been off for many days for some reason.
- 21:18 Ryan_Lane: rebooting all bastions
- 19:25 mutante: added new project "math" (Test to render TeX to MathML within the MW Framework)
- 11:01 petan: +DixonD
- 20:00 mutante: killing and restarting nrpe-server on all instances (bug 46026)
- 02:04 Ryan_Lane: rebooting bastion-restricted
- 15:10 hashar: puppet1.pmtpa.wmflabs is dead.
- 09:57 hashar: relaying Ryan: he restarted ldap on virt0 (was hung after server restart). nscld was properly falling back to virt1000 but ldap was stuck there too. DNS got restarted.
- 10:03 petan: Failed to add Fox Wilson to bastion.
- 10:03 petan: giving access to Fox Wilson
- 20:17 petan: giving access to Merlijn van Deen
- 08:34 petan: testing
- 08:34 petan: testing
- 07:09 Ryan_Lane: rebooting bastion-restricted.wmflabs.org
- 07:07 Ryan_Lane: rebooting bastion.wmflabs.org
- 10:16 hashar: /home/ is filling up again :(
- 16:07 paravoid: disabled hazard-sj's cron, as it was accidentally DoSing bastion
- 16:21 petan: test
- 20:33 hashar: DNS issue fixed by adding a LDAP failover on the NS servers bug 40825
- 20:02 mutante: adding new member Nemo_bis
- 13:32 Hydriz: Added Huji to the project. Requested for shell access, but is already in the shell group.
- 14:48 mutante: labs now has > 100 projects - We just created project #101.
- 14:44 mutante: created new project metavidwiki with gchriss as admin
- 13:01 mutante: labs.wikimedia.org is now a redirect to labsconsole for your convenience
- 09:41 mutante: - re-adding unassigned IP 18.104.22.168 to virt2 pool
- 13:12 mutante: deleting puppet variable mariadb_version
- 13:04 mutante: deleted mariadb classes from nova puppet groups
- 17:31 Thehelpfulone: added Kangaroopower to the project
- 11:38 mutante: adding Kai Nissen (WMFDE) to bastion
- 14:46 mutante: creating new project mediawiki-custom-de
- 15:29 Thehelpfulone: also added Sven_Manguard a few days ago
- 15:25 Thehelpfulone: added Tanvir to the object a few hours ago
- 16:08 mutante: added new member mschon to grant him shell access to wikistats instance
- 14:31 mutante: added myself to admin roles to see if that lets me login to debug ssh key for diederik, but it doesnt
- 03:44 mutante: - getting my Connection closed after the "If you are having access problems.." message. just me or all ?
- 02:01 Ryan_Lane: restricted login on bastion-restricted1 to people in the ops group
- 02:01 Ryan_Lane: restricted login on bastion1 to people in the bastion group, and am disallowing ops
- 10:36 petan|wk: new member IWorld
- 18:07 maplebed: added awjrichards to the bastion project
- 18:06 maplebed: creating awjr's labs account by running add-labs-user --wikiname=awjrichards --mail="email@example.com" awjrichards
- 16:43 petan|wk: given access to aude to labs
- 15:10 mutante: created project "translation-memory" for Nikerabbit
- 15:01 mutante: added member nikerabbit
- 14:43 mutante: added nikerabbit as a labs user, using add-labs-user on virt0
- 17:39 mutante: still asks for reboot
- 17:38 petan: petan fixed me
- 20:36 Ryan_Lane: temporarily removing Hydriz
- 21:38 Ryan_Lane: installing git-review via pip
- 21:38 Ryan_Lane: installing python-pip
- 12:21 mutante: - bastion1 asks for reboot
- 21:56 Ryan_Lane: installed tofrodos and python-paramiko on bastion1
- 19:16 mutante: added new member Coren
- 02:58 Ryan_Lane: test