Nova Resource:Paws/SAL

From Wikitech
Jump to navigation Jump to search

2020-05-26

  • 22:34 bstorm_: restored the deployment for maintain-kubeusers so anyone added to the paws.admin group will have admin on the cluster now that the bug is fixed T211096 T246059
  • 22:05 bstorm_: temporarily deleted the deployment for maintain-kubeusers pending patch to fix context creation for new admin accounts T211096 T246059
  • 22:04 bstorm_: created paws-focused PodSecurityPolicies and the prod namespace in the new cluster T211096
  • 22:03 bstorm_: created paws.admin group and kubernetes admin accounts on the new k8s cluster T211096 T246059
  • 18:29 bstorm_: bootstrapped the new control plane nodes T211096
  • 15:27 bstorm_: updated profile::wmcs::kubeadm::kubernetes_version to 1.16.10 for cluster init T211096

2020-05-21

  • 23:04 bstorm_: added profile::wmcs::kubeadm::k8s::encryption_key and profile::wmcs::kubeadm::k8s::node_token to labs/private T211096
  • 14:53 bstorm_: adding the hiera values to horizon for bootstrapping k8s T211096
  • 14:39 arturo: point record `k8s.svc.paws.eqiad1.wikimedia.cloud` to `172.16.1.186` (which is paws-k8s-control-1, for the initial bootstrap) (T211096)
  • 12:48 arturo: created record `k8s.svc.paws.eqiad1.wikimedia.cloud` pointing to `172.16.0.191` (which is paws-k8s-haproxy-1) (T211096)
  • 12:34 arturo: created and transferred DNS zone `svc.paws.eqiad1.wikimedia.cloud` (T211096)

2020-05-20

  • 22:35 bstorm_: created paws-k8s-worker-1/2/3/4 T211096
  • 22:12 bstorm_: created paws-k8s-haproxy-1/2 with antiaffinity group T211096
  • 21:36 bstorm_: created paws-k8s-control-1/2/3 with appropriate sec group and server group T211096
  • 18:59 bstorm_: created anti-affinity group "controlplane" T211096
  • 16:38 bstorm_: deleting the old shut-down VMs from the last effort to rebuild paws T211096
  • 16:36 bstorm_: cleaned up the old DNS entries for the external LBs that have been off for a year

2020-03-20

  • 14:03 jeh: upgrade paws-puppetmaster-01 to v5 T241719

2020-02-14

  • 21:31 andrewbogott: restarting paws-puppetmaster-01 so its clients can connect

2020-01-09

  • 18:06 bstorm_: rebooting tools-paws-master-01 T242353
  • 14:28 chicocvenancio: shutdown unused instances

2019-12-13

  • 00:27 bstorm_: rebooting the paws master since it is in a bad state after the openstack maintenance as well.

2019-11-01

  • 21:15 Krenair: Updated paws-apiserver.wmflabs.org A record list to remove 172.16.2.151 which is not allocated to any instance. The other two A records point to valid instances in the paws project.

2019-10-23

  • 09:03 arturo: paws-master-01/03 and a couple of other servers are down because hypervisor is rebooting

2019-10-14

  • 22:32 bd808: Removed project member "Afrodric". Looks like someone added accidentally when trying to make aborrero as project member
  • 22:31 bd808: Added Krenair as project member

2019-05-18

  • 11:13 chicocvenancio: point paws-proxy-02 to tools-paws-worker-1006 on paws-deploy-hook hostname (T218380)

2019-04-26

2019-04-16

  • 17:15 chicocvenancio: move paws-proxy-02 reload nginx
  • 17:07 chicocvenancio: move paws-proxy-02 to point to tools-paws-worker-1006 for upcoming master move

2019-03-27

  • 23:46 chicocvenancio: moving paws host in `paws-proxy-02` back to `tools-paws-master-01` T219460
  • 22:10 chicocvenancio: moving paws host in `paws-proxy-02` to `tools-paws-worker-1005` T219460

2019-03-25

  • 14:12 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project (T211096)
  • 14:07 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project T211096
  • 13:54 gtirloni: created `paws.wmflabs.org` subdomain under `paws` project (T211096)

2019-03-15

  • 02:25 gtirloni: activated TLS termination using Let's Encrypt on paws-proxy-02
  • 02:25 gtirloni: removed webproxies and created new A records pointing directly to paws-proxy-02

2019-02-21

  • 09:22 gtirloni: upgraded and rebooted paws-proxy-02

2019-02-20

  • 15:00 andrewbogott: deleting the long-shut-down paws-proxy-01

2019-02-15

  • 01:28 bd808: Re-enabled PAWS vhost on paws-proxy-02

2019-02-14

  • 22:25 gtirloni: downtimed PAWS in Icinga
  • 22:16 gtirloni: Activated maintenance page on paws-proxy-02 nginx config

2019-02-13

  • 08:32 arturo: switch paws-proxy-02 puppetmaster to labs-puppetmaster.wikimedia.org

2019-01-24

  • 19:20 andrewbogott: shutting down paws-proxy-01
  • 19:11 chicocvenancio: moved config, ready to receive traffic on paws-proxy-02 T214613
  • 18:34 chicocvenancio: firing up paws-proxy-02 for T214613

2019-01-23

2018-10-25

  • 23:58 gtirloni: Started tools-paws-worker-1010 (T208006)

2018-08-03

  • 20:19 andrewbogott: deleting paws-master-01 and paws-node-1002; unused

2018-07-03

  • 22:49 bstorm_: added stricter image space reclaiming arguments to kubelet

2018-06-20

  • 17:39 chicocvenancio: edited paws-proxy-01 to pass http_x_forwarded_proto as it receives T197248

2018-05-04

  • 02:48 chicocvenancio: killed 25 pods with more than one hour inactivity through admin interface

2018-03-14

  • 21:49 chicocvenancio: updated k8s control plane, updating nodes to v1.9.4 for T189680

2018-02-23

  • 18:33 chicocvenancio: redirected tools.wmflabs.org/paws to paws.wmflabs.org and deleted old k8s ReplicationControllers (T188068)

2018-02-22

  • 22:11 chicocvenancio: (T175202) culler is running and killing pods as designed!
  • 21:13 chicocvenancio: jupyterhub updated to fix culler (T175202) culler already ran without 404
  • 17:43 chicocvenancio: manually ran culler inside hub container

2018-02-21

  • 17:03 chicocvenancio: deleted query-killer k8s deployment T187818

2018-02-16

  • 20:18 chicocvenancio: changed userhomes group for T185434 workarround

2018-02-15

  • 01:10 chicocvenancio: changed group of all userhome folders to tools.paws

2018-02-04

  • 12:21 chicocvenancio: changed group of all userhome folders to tools.paws

2017-12-19

  • 22:11 bd808: Killed tiller pod that was in crashloopbackoff

2017-09-28

  • 21:25 andrewbogott: server docker restart on paws-node-1002; disk is full and docker is holding open a lot of deleted files

2017-03-20

  • 21:25 andrewbogott: migrating paws-base-01 to labvirt1013

2016-05-10