Nova Resource:Account-creation-assistance/SAL

From Wikitech
Jump to navigation Jump to search

2021-01-07

  • 11:16 dcaro: removing custom mx hosts, as the global names are now resolvable again (T271322)

2021-01-06

  • 14:53 dcaro: manually configured mx servers to use wikimedia.cloud domain on project hiera (T271322)

2019-06-19

  • 20:35 Izhidez: apache2 restarted
  • 20:33 Izhidez: apache2 shutdown to deal with spam (742 line DB change)

2019-01-31

  • 12:05 arturo: VM instances accounts-appserver5, were stopped briefly due to issue in hypervisor (T215012)
  • 12:04 arturo: VM instances accounts-appserver5, were stopped briefly due to issue in hypervisor (T215012)

September 20

  • 14:00 andrewbogott: moving accounts-mwoauth to virt1005

January 15

  • 08:23 andrewbogott: rebooted accounts-puppetmaster

November 1

  • 06:33 FastLizard4: running apt-get updates and reboot accounts-application

May 30

  • 01:28 FastLizard4: `apt-get update && apt-get dist-upgrade -y` on all instances

April 4

  • 01:34 FastLizard4: Looks like the problem was caused by puppet not having been run in days, so the ldap config was out of date (along with the puppet config)
  • 01:25 FastLizard4: SSH on accounts-database is acting up again, will reboot and investigate

April 2

  • 00:24 FastLizard4: accounts-application requesting reboot, fulfilling
  • 00:23 FastLizard4: `apt-get update && apt-get upgrade -y` on all accounts-*
  • 00:18 FastLizard4: Unable to SSH in to accounts-database, issuing a reboot

March 18

  • 06:47 FastLizard4: Altered security group 'default' config, changed rule 'from 80 to 80 proto tcp ip 10.4.0.54/32' to 'from 80 to 80 proto tcp ip *10.4.0.0/21*' to fix issues with accessing web services on accounts-application

March 15

  • 08:08 FastLizard4: updated packages on all instances

March 14

  • 02:22 FastLizard4: apt-get update/upgrade on all instances

March 10

  • 12:03 FastLizard4: Opreations log, supplemental: This was a triumph; I'm making a note here, huge success, and that turning it off and on again still works!
  • 11:57 FastLizard4: Opreations log, stardate 45650.54: -application is now properly reporting to Icinga, but -database isn't. Giving -database the (re)boot to see if that helps.

March 8

  • 23:55 FastLizard4|away: Supplemental: Reboot has fixed connectivity issues and application is back up. DHCP is probable culprit.
  • 23:54 FastLizard4|away: Supplemental: accounts-application is the instance that is unexpectedly down
  • 23:53 FastLizard4|away: is unexpectedly down, attempting a reboot to recover. Will investigate further when I get home.
  • 09:36 FastLizard4: Fixed incinga reporting by `sudo cp -R /etc/nagios/* /etc/icinga/ && sudo killall nrpe && sudo /etc/init.d/nagios-nrpe-server start` on -application and -database.
  • 08:47 FastLizard4: Nope, reboots did not fix the errors. Interestingly, only application and database are showing errors (for load, disk, RAM, numprocs, and dpkg): "CHECK_NRPE: Error - Could not complete SSL handshake." Puppetmaster is reading all-green.
  • 08:35 FastLizard4: Going to try rebooting all systems to see if that fixes the NRPE problems.

March 6

  • 03:11 FastLizard4: All servers requesting reboots, will provide
  • 03:06 FastLizard4: Running apt-get upgrade on all server

February 25

  • 05:48 FastLizard4: THe servers demand reboots, and reboots they shall receive

February 21

  • 10:11 FastLizard4: All instances need reboots after `apt-get` upgrade. Initiating now.

February 17

  • 03:48 FastLizard4: Scratch last log entry, error fixed by using `sudo puppetca --sign "i-000005c5.pmtpa.wmflabs"`
  • 03:41 FastLizard4: Getting strange puppet errors related to SSL on accounts-puppetmaster, going to try rebooting.

February 16

  • 10:43 FastLizard4: `apt-get update && apt-get upgrade` run on all instances

February 13

  • 07:01 FastLizard4: `apt-get upgrade` and reboot on accounts-database

February 12

  • 23:45 FastLizard4: Installing package updates and rebooting accounts-application. Also got puppet running again by removing generic::packages::git-core class.
  • 23:23 FastLizard4: Setting mysql_server_bind_address to * for accounts-database
  • 23:03 FastLizard4: Restarting accounts-puppetmaster to apply updates to packages
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Nova_Resource:Account-creation-assistance/SAL&oldid=1892608"