Difference between revisions of "Nova Resource:Deployment-prep/SAL/Archive 1"

From Wikitech
Jump to navigation Jump to search
(petrb: making syslog on apaches be /data/project/apaches_log (labs-logs-bottie))
m (→‎May 3: make myself make sense ;))
(3 intermediate revisions by one other user not shown)
Line 1: Line 1:
  +
=== May 3 ===
  +
* 04:48 jeremyb: [deployment-dbdump] (that was to address complaints about beta simplewiki appearing in #cvn-simplewikis on freenode)
  +
* 04:47 jeremyb: [deployment-dbdump] changed all refs to IPs of prod hosts nfs-home and ekrem to be deployment-feed instead. and commited that to the local repo. (again not pushed anywhere yet)
  +
* 04:44 jeremyb: [deployment-dbdump] did a checkpoint `git commit -a` on deploymentprep-conf (/usr/local/apache/common) (locally not pushed anywhere) because there were lots of changes on disk but not in the repo. but didn't add any new files to the repo. (so there's still stuff reported uncommited by `git status`)
  +
 
=== May 2 ===
 
=== May 2 ===
 
* 15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log
 
* 15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log

Revision as of 04:56, 3 May 2012

May 3

  • 04:48 jeremyb: [deployment-dbdump] (that was to address complaints about beta simplewiki appearing in #cvn-simplewikis on freenode)
  • 04:47 jeremyb: [deployment-dbdump] changed all refs to IPs of prod hosts nfs-home and ekrem to be deployment-feed instead. and commited that to the local repo. (again not pushed anywhere yet)
  • 04:44 jeremyb: [deployment-dbdump] did a checkpoint `git commit -a` on deploymentprep-conf (/usr/local/apache/common) (locally not pushed anywhere) because there were lots of changes on disk but not in the repo. but didn't add any new files to the repo. (so there's still stuff reported uncommited by `git status`)

May 2

  • 15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log
  • 11:52 labs-logs-bottie: petrb: changed sudo policies on web to test if puppet override it
  • 11:52 labs-logs-bottie: j: add transcoding settings to CommonSettings.php again
  • 11:46 labs-logs-bottie: petrb: purged stuff on transcoding and freed some 119468kb
  • 11:08 labs-logs-bottie: hashar: install dsh package on deployment-dbdump
  • 09:54 labs-logs-bottie: hashar: /usr/local/apache/conf is now an independant git repository

April 30

  • 13:31 labs-logs-bottie: petrb: rebooting web5
  • 13:27 labs-logs-bottie: petrb: web4 reboot for same reason
  • 13:26 labs-logs-bottie: petrb: same for web
  • 13:24 labs-logs-bottie: petrb: rebooting web3 broken /data/project/
  • 11:57 labs-logs-bottie: petrb: fixed transcoding
  • 11:28 j^: reboot deployment-transcoding(i-00000105)

April 25

  • 17:01 labs-logs-bottie: j: Changing uid and group of apache user from 48 to 33 to match www-data on web3,web4,web5
  • 16:44 Platonides: With the uid change to deployment-web, it is now writing into /data/project/errors.log
  • 16:42 hashar: Apaches no more log anything. This is because rsyslog sends logs to a blackhole :-D
  • 16:27 Platonides: Changing uid and group of apache user from 48 to 33 to match www-data
  • 15:20 hashar: deleting testswarmmysqlconf , it is of no use :-/
  • 14:43 hashar: Creating temporary instance to test a MySQL puppet snippet
  • 13:01 labs-logs-bottie: hashar: I am a hero
  • 13:00 labs-logs-bottie: petrb: test
  • 12:24 hashar: added a basic puppet skeleton in manifests/labs/beta/ with https://gerrit.wikimedia.org/r/5790 (test branch)
  • 11:39 hashar: manually purged debian package `ack`, installed `ack-grep`
  • 11:22 hashar: puppet finished migration of web{3,4,5} to apache::service
  • 11:19 hashar: changed squid visible name to squid001.beta.wmflabs.org
  • 11:02 hashar: migrate web{3,4,5} from webserver::php5 to apaches::service
  • 10:47 hashar: Cleaning out squid peers list
  • 10:11 hashar: made /etc/ a git repository on deployment-squid and committed existing /etc/squid/
  • 10:03 hashar: adding generic::packages::git-core on deployment-squid so we can track

April 24

  • 23:21 labs-logs-bottie: petrb: disabled $wmgEnableCaptcha
  • 18:54 hashar: wrap -dbdump motd to 80 chars
  • 15:42 hashar: deployement-web host does work! :-]
  • 15:41 hashar: made deployement-web host a wikimedia-task-appserver , add to create some apache2 configuration placeholder. Apache2 does launch but it is not working though (timeout)
  • 15:40 petan|wk: I told hashar to log stuff, if he won't, slap him
  • 13:18 hashar: added apache::service on deployment-web host
  • 12:18 labs-logs-bottie: petrb: moved the log file storage to gluster
  • 12:18 labs-logs-bottie: petrb: updated git and commited all changes

March 21

  • 03:22 mutante: mysqld on deployment-sql is stopped - did not start it though after i heard petan is working on corrupted db's
  • 03:06 mutante: added myself as a member just to see the instance names and check for the sql server...

March 20

  • 16:16 labs-logs-bottie: petrb: it seems that corruption of db is worse than I expected, need to restore backup old few months
  • 16:12 labs-logs-bottie: petrb: mysql is back up
  • 15:41 labs-logs-bottie: petrb: getting sql server down I found a bunch of corrupted db's, rollback is necessary
  • 15:41 labs-logs-bottie: j: install php-pear on deployment-web3/4/5 required by TMH

March 19

  • 08:18 labs-logs-bottie: root: restoring sql tables from backup

March 15

  • 20:17 labs-logs-bottie: root: restored ok
  • 20:14 labs-logs-bottie: root: restoring database from backup
  • 10:16 labs-logs-bottie: petrb: failed auth on db server reboot was required
  • 09:53 labs-logs-bottie: petrb: scheduling auto replication of sql server

March 14

  • 14:37 labs-logs-bottie: petrb: switching en.wikipedia to older previous v

March 11

  • 22:14 Damianz: Increased nofile on deployment-squid and added max_filedesc option to squid config. Also installed squidclient.
  • 04:36 Ryan_Lane: also deployment-web3
  • 04:35 Ryan_Lane: also deployment-web
  • 04:34 Ryan_Lane: make that deployment-web5
  • 04:34 Ryan_Lane: rebooting deployment-web, it OOM'd

March 9

  • 14:12 labs-logs-bottie: petrb: rebooting -nfs
  • 13:52 labs-logs-bottie: root: updated apt on webs1
  • 13:34 j^: add ppa:j/timedmediahandler and install ffmpeg on web3 and web5

March 6

  • 22:21 labs-logs-bottie: petrb: some instances will need to reboot, however site seems to be ok now
  • 22:09 labs-sexy-bottie: petrb: updating svn
  • 22:07 labs-sexy-bottie: petrb: fixed squid a bit
  • 15:23 labs-sexy-bottie: petrb: test
  • 14:53 labs-sexy-bottie: root: disabling bot for a while
  • 03:11 Andrew: facepalm: apparently all reboots are failing, so this will be down until Ryan brings it all back up tomorrow
  • 02:57 Andrew: rebooting a few hosts, there is something seriously wrong with fetching resources at the moment

March 5

  • 15:48 labs-sexy-bottie: petrb: temporary disabled ssl server
  • 15:48 labs-sexy-bottie: petrb: reconfigured squid
  • 15:36 labs-sexy-bottie: petrb: restarted servers
  • 15:24 labs-sexy-bottie: petrb: temporary changed code of localsettings to debug site
  • 15:20 labs-sexy-bottie: petrb: fixed broken memc :o
  • 15:04 labs-sexy-bottie: petrb: inserted new wiki to sul
  • 15:01 labs-sexy-bottie: petrb: please ignore some of the previous lines in log we were just testing bot
  • 14:58 labs-sexy-bottie: petrb: updated live
  • 14:58 labs-sexy-bottie: petrb: meh
  • 14:36 labs-sexy-bottie: petrb: created a new log system, just type log message to log your change on prep
  • 14:35 labs-sexy-bottie: petrb: this is test :o

March 4

  • 12:00 Andrew: Finished deployment of het deploy, added a new ee.

March 1

  • 16:35 Platonides: Installed dpkg-dev on deployment-dbdump
  • 16:03 Platonides: Installed joe on deployment-dbdump
  • 16:03 petan|wk: platonides needs to check the project name

February 27

  • 08:20 petan: creating 2 more web servers to handle load
  • 08:19 petan: rebooting both web servers, starting with web1

February 23

  • 08:52 petan|wk: fixing the squid

February 22

  • 01:03 Ryan_Lane: reconfiguring the web server instances to remove puppet classes that no longer exist

February 17

  • 02:34 Andrew: Moving /usr/local/apache/common/live to /usr/local/apache/common/live-hom and symlinking live to live-hom
  • 01:56 Andrew: running afl_rev_id patch on all wikis
  • 01:52 Andrew: installing ack (source code search tool) on dbdump
  • 00:59 petan: if anything is broken, it was me
  • 00:51 petan: I broke it!
  • 00:27 petan: switched to HEAD

February 16

  • 10:11 j^: add video/webm to /etc/mime.types on web/webs1/web2

February 13

  • 10:08 petan|wk: removing the puppetized memcached
  • 08:54 petan|wk: removing some extensions from config which are missing in latest branch

January 30

  • 18:46 petan: configuring some boxes for cluster to handle high load

January 29

  • 01:13 hexmode: oom reboot -web

January 27

  • 13:30 j^: install upstart script /etc/init/timedmediahandler.conf on deployment-transcoding and start service
  • 13:05 j^: touch /etc/wikimedia-image-scaler on deployment-transcoding; transcoding needs more wgMaxShellMemory too
  • 12:47 Platonides: updated /usr/local/apache/common/live/extensions/TimedMediaHandler to r110117 per j^request
  • 10:11 j^: add-apt-repository ppa:j/timedmediahandler and update ffmpeg on deployment-web to support frame extraction from WebM videos
  • 06:35 j^: update ffmpeg on deployment-transcoding (new security release from ppa)

January 26

  • 00:00 petan: configured new firewall rule irc

January 25

  • 23:52 petan: linked /usr/local/apache/common-local to /usr/local/apache/common
  • 23:06 petan: updating svn
  • 22:28 petan: reverted unlogged changes made to config which broke whole site
  • 10:03 j^: installed ffmpeg on deployment-web (required by TMH to extract stills)

January 24

  • 20:52 petan: created db user oren and new database for temporary wiki
  • 13:53 petan|wk: reconfigured new instance and fixed some issues on puppet, no logs in sal regarding it
  • 00:51 hexmode: svn up * updatedata

January 23

  • 19:31 hexmode: restart memcache on nfs-memc
  • 19:06 hexmode: aptitude update deployment-web

January 22

  • 09:16 petan: configured nfs to listen for backup server
  • 01:01 petan: configured firewall for backup instance
  • 00:45 petan: creating a backup instance in -prepbackup project for online backup of mysql from deployment project + fs backup
  • 00:30 petan: updating /live to head
  • 00:21 petan: installed timedmediahandler (trunk) to commons

January 16

  • 23:41 hexmode: to solve the trusted XFF problem, I installed tinycdb and created an 0 length file in the right place
  • 20:35 Ryan_Lane: released unused IP address from project

January 15

  • 15:45 petan: ran live/extensions/TrustedXFF/generate.php
  • 11:20 petan: updated to latest head all wikis

January 14

  • 18:49 petan: enabled global blocking
  • 18:21 johnduhart: Removed myself from the project.
  • 14:32 petan: separated common to own deployment file
  • 14:30 hexmode: enabled webfonts for mywiki properly in IntialiseSettingsDeploy.php
  • 14:25 johnduhart: Updated wmf-config/InitialiseSettings.php from production
  • 14:25 johnduhart: Reverted change to wmf-config/InitialiseSettings.php
  • 14:24 hexmode: enabled webfonts for mywiki

January 13

  • 21:34 petan: assigning new dns
  • 21:33 petan: moved deployment to beta.wmf...
  • 13:07 petan|w: installed jdk on search

January 12

  • 18:49 petan: installed all requested sw on search
  • 18:46 petan: mounted conf files
  • 18:38 petan: installed updates on new instances and rebooting it
  • 06:48 Ryan_Lane: added nfs mounts to the fstab for deployment-web
  • 06:47 Ryan_Lane: remounted /mnt/upload on deployment-web as nfs rather than nfs4
  • 06:47 Ryan_Lane: modified export options on deployment-nfs-memc; removed nfs4 specific options, and removed other options not necessary for our environment.
  • 00:38 johnduhart: Unmounted /mnt/export from /tmp on -web

January 11

January 10

  • 23:06 petan: checks done
  • 23:04 petan: disabled sql for fs checks
  • 22:30 petan: created nfs:/mnt/export/backup use it for all files which aren't versioned
  • 21:44 petan: deleting big db's expect db lags ^^
  • 21:35 petan: maintenance on simple
  • 21:17 petan: created squid box
  • 19:05 petan: tweaked memcached and flushed cache
  • 16:48 johnduhart: installing wikimedia-task-appserver on -web
  • 16:29 johnduhart: live and extensions recheckedout into a new folder
  • 16:04 johnduhart: Site broken, currently recreating live folder
  • 12:27 johnduhart: Running updatedata (very slowly)
  • 12:19 johnduhart: Fixed wmf-config permissions on -nfs-memc
  • 05:32 johnduhart: thumbnails now working
  • 05:30 johnduhart: Installed imagemagick on web
  • 05:11 johnduhart: Adding apache config for upload.deployment.wmflabs.org
  • 05:11 johnduhart: Made a quick stab at upload config
  • 05:00 johnduhart: Mounted that export onto -web
  • 05:00 johnduhart: Created nfs export /mnt/upload on deployment-nfs-memc
  • 04:37 johnduhart: unmounted deployment-nfs-memc:/mnt/export on /mnt from deployment-web
  • 03:32 johnduhart: Last update solves an issue where CentralAuth would make 70+ queries per page
  • 03:31 johnduhart: Updated databases
  • 03:26 johnduhart: svn up'd live
  • 03:04 johnduhart: interwiki now works

January 9

  • 21:29 johnduhart: Updated databases
  • 21:29 johnduhart: Created test.wikimedia
  • 21:25 petan: reconfigured global permissions
  • 20:38 petan: created hi wiki + de wiki
  • 20:00 petan: creating commons, de wiki, en_wiktionary etc. etc...
  • 18:57 petan: configured ip back to -web and removed temporary ip
  • 18:53 petan: disasociated 208.80.153.219
  • 18:50 petan: turned off -test definitely and reconfiguring ip
  • 18:47 petan: moved all stuff -web reconfiguring IP
  • 18:18 petan: moving configuration of apache to web
  • 16:40 petan|work: reconfigured apache on test
  • 16:15 johnduhart: svn up /usr/local/apache/common/live
  • 15:59 johnduhart: Ran update on metawiki
  • 15:58 johnduhart: Recreated metawiki enwiki enwikibooks
  • 15:57 johnduhart: Imported centralauth
  • 15:55 johnduhart: Dropped all wikis except simplewiki
  • 15:55 johnduhart:
  • 15:53 petan|work: restarted memcached
  • 15:24 johnduhart: Created enwiki and enwikibooks
  • 15:20 johnduhart: Restarted memcached
  • 15:17 johnduhart: Created simplewiki
  • 15:04 petan|work: restarted memcached
  • 15:02 johnduhart: Ran update.php on metawiki
  • 14:59 johnduhart: Creating metawiki
  • 14:55 petan|work: disabled current site
  • 14:55 johnduhart: Creating centralauth db
  • 14:52 johnduhart: DROPing new configuration tables, will recreate
  • 14:38 johnduhart: Forget the metawiki dump
  • 14:09 petan|work: created backup of broken db of meta and replaced it with auth db
  • 13:52 petan|work: test is done, restored test SUL to previous state
  • 13:43 petan|work: created backup of central auth and replace the testing SUL with current data, merged with current SUL so that we can use same logins on all sites
  • 13:19 petan|work: updated svn
  • 12:44 johnduhart: Ran update.php on metawiki
  • 12:32 johnduhart: Created enwikibooks http://en.wikibooks.deployment.wmflabs.org/wiki/Main_Page
  • 08:12 johnduhart: Starting import of metawiki
  • 07:37 johnduhart: This only affects my new configuration though
  • 07:36 johnduhart: WARNING: some how new users are showing up on some feed outside of labs, and was picked up on a monitoring robot. wtf.
  • 07:15 johnduhart: Importing simplewikibooks
  • 07:10 johnduhart: SiteMatrix is now working http://meta.wikimedia.deployment.wmflabs.org/wiki/Special:SiteMatrix
  • 07:09 johnduhart: Created simple wikibooks http://simple.wikibooks.deployment.wmflabs.org/
  • 07:09 johnduhart: Adding wikibooks configuration
  • 06:31 johnduhart: Installing git on deployment-test
  • 06:23 johnduhart: Created simplewiki http://simple.wikipedia.deployment.wmflabs.org/wiki/Main_Page
  • 05:50 johnduhart: Downloaded metawiki dump ont dbdump, extracting now. poor labs.
  • 05:36 johnduhart: Recreated hwiki, had a bad JohnTest account. Central auth now works fully
  • 05:15 johnduhart: Running update.php on hiwiki
  • 05:14 johnduhart: Importing prefstats table to hiwiki
  • 04:41 johnduhart: Centralauth is working :)
  • 04:28 johnduhart: Imported db schema to centralauth
  • 04:25 johnduhart: Created centralauth database centralauth
  • 04:18 johnduhart: Created metawiki http://meta.wikimedia.deployment.wmflabs.org/
  • 04:18 johnduhart: http://meta.wikimedia.deployment.wmflabs.org/
  • 04:09 johnduhart: Created meta docroot and added favicons to meta and wp
  • 04:08 johnduhart: Adding remnant.conf to httpd.conf
  • 04:07 johnduhart: Adding metawiki apache config to /usr/local/apache/conf/remnant.conf
  • 04:03 johnduhart: Lot of configuration done, much tuning to come. Missing.php now works http://nope.wikipedia.deployment.wmflabs.org/wiki/Main_Page
  • 01:24 johnduhart: Installed mysql-client on deployment-test

January 8

  • 23:45 johnduhart: Installed php5-cli on deployment-test
  • 23:21 johnduhart: Enabled rewrite rules on deployment-test
  • 20:39 petan: created new group for /mnt/www
  • 19:33 petan: clearing cache
  • 18:47 hexmode: set up relative paths for config files
  • 16:45 petan: moved memcached to deployment-nfs-memc and update global
  • 14:04 petan: created temporary memcached instance for data
  • 12:17 petan: restarted apache to fix some problem with config
  • 00:11 hexmode: set config for timeline
  • 00:11 hexmode: "apt-get install ploticus ttf-freefont" for timeline
  • 00:04 jeremyb: booting memcached fixed it
  • 00:02 jeremyb: booting memcache in case that fixes interwiki
  • 00:02 jeremyb: [auth] added bugzilla to interwiki.sql and ran it, doesn't seem to be working

January 7

  • 23:55 hexmode: Fix path for PoolCounter: PoolCounter.php -> PoolCounterClient.php
  • 23:52 hexmode: Fix path for OAI: OAI.php -> OAIRepo.php
  • 23:31 hexmode: set up git for /var/www/global
  • 23:08 petan: disabled LQT on other wikis
  • 23:07 petan: changed configuration of ajax
  • 23:07 petan: fixed InitialiseSetting
  • 22:42 petan: updated all wikis to latest head
  • 22:36 petan: disabled LQT because it's broken
  • 22:34 petan: ran update on auth db
  • 22:10 petan: removed puppetized memcached, because its configuration suck

January 5

  • 20:26 petan: importing MW ns full history to en_wikipedia
  • 16:48 petan|w: created instance for dumps, current instance is overloaded
  • 15:53 petan|w: import now running using mwimport

January 4

  • 20:27 petan: opened port 80 for wide net
  • 20:26 petan: registered deployment.wmflabs.org
  • 20:24 petan: allocated ip208.80.153.215
  • 20:22 mutante: raised floating IP quota to 1
  • 16:10 petan|work: create instances for apache and mysql
  • 16:08 petan|work: configured firewall for webserver
  • 16:06 mutante: added members MarkAHershberger & Petrb - added them to sysadmin and netadmin roles
  • 16:02 mutante: added new project deployment-prep for hexmode and petan
  • 16:02 mutante: added new project deployment-prep for hexmode and petan