Nova Resource:Deployment-prep/SAL/Archive 1: Difference between revisions

Browse history interactively

← Older edit Newer edit →

Content deleted Content added

VisualWikitext

Inline

Revision as of 04:56, 3 May 2012

May 3

04:48 jeremyb: [deployment-dbdump] (that was to address complaints about beta simplewiki appearing in #cvn-simplewikis on freenode)
04:47 jeremyb: [deployment-dbdump] changed all refs to IPs of prod hosts nfs-home and ekrem to be deployment-feed instead. and commited that to the local repo. (again not pushed anywhere yet)
04:44 jeremyb: [deployment-dbdump] did a checkpoint `git commit -a` on deploymentprep-conf (/usr/local/apache/common) (locally not pushed anywhere) because there were lots of changes on disk but not in the repo. but didn't add any new files to the repo. (so there's still stuff reported uncommited by `git status`)

May 2

15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log
11:52 labs-logs-bottie: petrb: changed sudo policies on web to test if puppet override it
11:52 labs-logs-bottie: j: add transcoding settings to CommonSettings.php again
11:46 labs-logs-bottie: petrb: purged stuff on transcoding and freed some 119468kb
11:08 labs-logs-bottie: hashar: install dsh package on deployment-dbdump
09:54 labs-logs-bottie: hashar: /usr/local/apache/conf is now an independant git repository

April 30

13:31 labs-logs-bottie: petrb: rebooting web5
13:27 labs-logs-bottie: petrb: web4 reboot for same reason
13:26 labs-logs-bottie: petrb: same for web
13:24 labs-logs-bottie: petrb: rebooting web3 broken /data/project/
11:57 labs-logs-bottie: petrb: fixed transcoding
11:28 j^: reboot deployment-transcoding(i-00000105)

April 25

17:01 labs-logs-bottie: j: Changing uid and group of apache user from 48 to 33 to match www-data on web3,web4,web5
16:44 Platonides: With the uid change to deployment-web, it is now writing into /data/project/errors.log
16:42 hashar: Apaches no more log anything. This is because rsyslog sends logs to a blackhole :-D
16:27 Platonides: Changing uid and group of apache user from 48 to 33 to match www-data
15:20 hashar: deleting testswarmmysqlconf , it is of no use :-/
14:43 hashar: Creating temporary instance to test a MySQL puppet snippet
13:01 labs-logs-bottie: hashar: I am a hero
13:00 labs-logs-bottie: petrb: test
12:24 hashar: added a basic puppet skeleton in manifests/labs/beta/ with https://gerrit.wikimedia.org/r/5790 (test branch)
11:39 hashar: manually purged debian package `ack`, installed `ack-grep`
11:22 hashar: puppet finished migration of web{3,4,5} to apache::service
11:19 hashar: changed squid visible name to squid001.beta.wmflabs.org
11:02 hashar: migrate web{3,4,5} from webserver::php5 to apaches::service
10:47 hashar: Cleaning out squid peers list
10:11 hashar: made /etc/ a git repository on deployment-squid and committed existing /etc/squid/
10:03 hashar: adding generic::packages::git-core on deployment-squid so we can track

April 24

23:21 labs-logs-bottie: petrb: disabled $wmgEnableCaptcha
18:54 hashar: wrap -dbdump motd to 80 chars
15:42 hashar: deployement-web host does work! :-]
15:41 hashar: made deployement-web host a wikimedia-task-appserver , add to create some apache2 configuration placeholder. Apache2 does launch but it is not working though (timeout)
15:40 petan|wk: I told hashar to log stuff, if he won't, slap him
13:18 hashar: added apache::service on deployment-web host
12:18 labs-logs-bottie: petrb: moved the log file storage to gluster
12:18 labs-logs-bottie: petrb: updated git and commited all changes

March 21

03:22 mutante: mysqld on deployment-sql is stopped - did not start it though after i heard petan is working on corrupted db's
03:06 mutante: added myself as a member just to see the instance names and check for the sql server...

March 20

16:16 labs-logs-bottie: petrb: it seems that corruption of db is worse than I expected, need to restore backup old few months
16:12 labs-logs-bottie: petrb: mysql is back up
15:41 labs-logs-bottie: petrb: getting sql server down I found a bunch of corrupted db's, rollback is necessary
15:41 labs-logs-bottie: j: install php-pear on deployment-web3/4/5 required by TMH

March 19

08:18 labs-logs-bottie: root: restoring sql tables from backup

March 15

20:17 labs-logs-bottie: root: restored ok
20:14 labs-logs-bottie: root: restoring database from backup
10:16 labs-logs-bottie: petrb: failed auth on db server reboot was required
09:53 labs-logs-bottie: petrb: scheduling auto replication of sql server

March 14

14:37 labs-logs-bottie: petrb: switching en.wikipedia to older previous v

March 11

22:14 Damianz: Increased nofile on deployment-squid and added max_filedesc option to squid config. Also installed squidclient.
04:36 Ryan_Lane: also deployment-web3
04:35 Ryan_Lane: also deployment-web
04:34 Ryan_Lane: make that deployment-web5
04:34 Ryan_Lane: rebooting deployment-web, it OOM'd

March 9

14:12 labs-logs-bottie: petrb: rebooting -nfs
13:52 labs-logs-bottie: root: updated apt on webs1
13:34 j^: add ppa:j/timedmediahandler and install ffmpeg on web3 and web5

March 6

22:21 labs-logs-bottie: petrb: some instances will need to reboot, however site seems to be ok now
22:09 labs-sexy-bottie: petrb: updating svn
22:07 labs-sexy-bottie: petrb: fixed squid a bit
15:23 labs-sexy-bottie: petrb: test
14:53 labs-sexy-bottie: root: disabling bot for a while
03:11 Andrew: facepalm: apparently all reboots are failing, so this will be down until Ryan brings it all back up tomorrow
02:57 Andrew: rebooting a few hosts, there is something seriously wrong with fetching resources at the moment

March 5

15:48 labs-sexy-bottie: petrb: temporary disabled ssl server
15:48 labs-sexy-bottie: petrb: reconfigured squid
15:36 labs-sexy-bottie: petrb: restarted servers
15:24 labs-sexy-bottie: petrb: temporary changed code of localsettings to debug site
15:20 labs-sexy-bottie: petrb: fixed broken memc :o
15:04 labs-sexy-bottie: petrb: inserted new wiki to sul
15:01 labs-sexy-bottie: petrb: please ignore some of the previous lines in log we were just testing bot
14:58 labs-sexy-bottie: petrb: updated live
14:58 labs-sexy-bottie: petrb: meh
14:36 labs-sexy-bottie: petrb: created a new log system, just type log message to log your change on prep
14:35 labs-sexy-bottie: petrb: this is test :o

March 4

12:00 Andrew: Finished deployment of het deploy, added a new ee.

March 1

16:35 Platonides: Installed dpkg-dev on deployment-dbdump
16:03 Platonides: Installed joe on deployment-dbdump
16:03 petan|wk: platonides needs to check the project name

February 27

08:20 petan: creating 2 more web servers to handle load
08:19 petan: rebooting both web servers, starting with web1

February 23

08:52 petan|wk: fixing the squid

February 22

01:03 Ryan_Lane: reconfiguring the web server instances to remove puppet classes that no longer exist

February 17

02:34 Andrew: Moving /usr/local/apache/common/live to /usr/local/apache/common/live-hom and symlinking live to live-hom
01:56 Andrew: running afl_rev_id patch on all wikis
01:52 Andrew: installing ack (source code search tool) on dbdump
00:59 petan: if anything is broken, it was me
00:51 petan: I broke it!
00:27 petan: switched to HEAD

February 16

10:11 j^: add video/webm to /etc/mime.types on web/webs1/web2

February 13

10:08 petan|wk: removing the puppetized memcached
08:54 petan|wk: removing some extensions from config which are missing in latest branch

January 30

18:46 petan: configuring some boxes for cluster to handle high load

January 29

01:13 hexmode: oom reboot -web

January 27

13:30 j^: install upstart script /etc/init/timedmediahandler.conf on deployment-transcoding and start service
13:05 j^: touch /etc/wikimedia-image-scaler on deployment-transcoding; transcoding needs more wgMaxShellMemory too
12:47 Platonides: updated /usr/local/apache/common/live/extensions/TimedMediaHandler to r110117 per j^request
10:11 j^: add-apt-repository ppa:j/timedmediahandler and update ffmpeg on deployment-web to support frame extraction from WebM videos
06:35 j^: update ffmpeg on deployment-transcoding (new security release from ppa)

January 26

00:00 petan: configured new firewall rule irc

January 25

23:52 petan: linked /usr/local/apache/common-local to /usr/local/apache/common
23:06 petan: updating svn
22:28 petan: reverted unlogged changes made to config which broke whole site
10:03 j^: installed ffmpeg on deployment-web (required by TMH to extract stills)

January 24

20:52 petan: created db user oren and new database for temporary wiki
13:53 petan|wk: reconfigured new instance and fixed some issues on puppet, no logs in sal regarding it
00:51 hexmode: svn up * updatedata

January 23

19:31 hexmode: restart memcache on nfs-memc
19:06 hexmode: aptitude update deployment-web

January 22

09:16 petan: configured nfs to listen for backup server
01:01 petan: configured firewall for backup instance
00:45 petan: creating a backup instance in -prepbackup project for online backup of mysql from deployment project + fs backup
00:30 petan: updating /live to head
00:21 petan: installed timedmediahandler (trunk) to commons

January 16

23:41 hexmode: to solve the trusted XFF problem, I installed tinycdb and created an 0 length file in the right place
20:35 Ryan_Lane: released unused IP address from project

January 15

15:45 petan: ran live/extensions/TrustedXFF/generate.php
11:20 petan: updated to latest head all wikis

January 14

18:49 petan: enabled global blocking
18:21 johnduhart: Removed myself from the project.
14:32 petan: separated common to own deployment file
14:30 hexmode: enabled webfonts for mywiki properly in IntialiseSettingsDeploy.php
14:25 johnduhart: Updated wmf-config/InitialiseSettings.php from production
14:25 johnduhart: Reverted change to wmf-config/InitialiseSettings.php
14:24 hexmode: enabled webfonts for mywiki

January 13

21:34 petan: assigning new dns
21:33 petan: moved deployment to beta.wmf...
13:07 petan|w: installed jdk on search

January 12

18:49 petan: installed all requested sw on search
18:46 petan: mounted conf files
18:38 petan: installed updates on new instances and rebooting it
06:48 Ryan_Lane: added nfs mounts to the fstab for deployment-web
06:47 Ryan_Lane: remounted /mnt/upload on deployment-web as nfs rather than nfs4
06:47 Ryan_Lane: modified export options on deployment-nfs-memc; removed nfs4 specific options, and removed other options not necessary for our environment.
00:38 johnduhart: Unmounted /mnt/export from /tmp on -web

January 11

23:02 hexmode: svn up live
03:19 johnduhart: Added production commons to ForeignFileRepos http://en.wikipedia.deployment.wmflabs.org/wiki/File:BradPittBAR08.jpg
02:28 johnduhart: Moved config to http://config.deployment.wmflabs.org/viewfile.php?file=CommonSettings.php
01:44 johnduhart: updated databases and interwiki
01:41 johnduhart: setup flagged revisions

January 10

23:06 petan: checks done
23:04 petan: disabled sql for fs checks
22:30 petan: created nfs:/mnt/export/backup use it for all files which aren't versioned
21:44 petan: deleting big db's expect db lags ^^
21:35 petan: maintenance on simple
21:17 petan: created squid box
19:05 petan: tweaked memcached and flushed cache
16:48 johnduhart: installing wikimedia-task-appserver on -web
16:29 johnduhart: live and extensions recheckedout into a new folder
16:04 johnduhart: Site broken, currently recreating live folder
12:27 johnduhart: Running updatedata (very slowly)
12:19 johnduhart: Fixed wmf-config permissions on -nfs-memc
05:32 johnduhart: thumbnails now working
05:30 johnduhart: Installed imagemagick on web
05:11 johnduhart: Adding apache config for upload.deployment.wmflabs.org
05:11 johnduhart: Made a quick stab at upload config
05:00 johnduhart: Mounted that export onto -web
05:00 johnduhart: Created nfs export /mnt/upload on deployment-nfs-memc
04:37 johnduhart: unmounted deployment-nfs-memc:/mnt/export on /mnt from deployment-web
03:32 johnduhart: Last update solves an issue where CentralAuth would make 70+ queries per page
03:31 johnduhart: Updated databases
03:26 johnduhart: svn up'd live
03:04 johnduhart: interwiki now works

January 9

21:29 johnduhart: Updated databases
21:29 johnduhart: Created test.wikimedia
21:25 petan: reconfigured global permissions
20:38 petan: created hi wiki + de wiki
20:00 petan: creating commons, de wiki, en_wiktionary etc. etc...
18:57 petan: configured ip back to -web and removed temporary ip
18:53 petan: disasociated 208.80.153.219
18:50 petan: turned off -test definitely and reconfiguring ip
18:47 petan: moved all stuff -web reconfiguring IP
18:18 petan: moving configuration of apache to web
16:40 petan|work: reconfigured apache on test
16:15 johnduhart: svn up /usr/local/apache/common/live
15:59 johnduhart: Ran update on metawiki
15:58 johnduhart: Recreated metawiki enwiki enwikibooks
15:57 johnduhart: Imported centralauth
15:55 johnduhart: Dropped all wikis except simplewiki
15:55 johnduhart:
15:53 petan|work: restarted memcached
15:24 johnduhart: Created enwiki and enwikibooks
15:20 johnduhart: Restarted memcached
15:17 johnduhart: Created simplewiki
15:04 petan|work: restarted memcached
15:02 johnduhart: Ran update.php on metawiki
14:59 johnduhart: Creating metawiki
14:55 petan|work: disabled current site
14:55 johnduhart: Creating centralauth db
14:52 johnduhart: DROPing new configuration tables, will recreate
14:38 johnduhart: Forget the metawiki dump
14:09 petan|work: created backup of broken db of meta and replaced it with auth db
13:52 petan|work: test is done, restored test SUL to previous state
13:43 petan|work: created backup of central auth and replace the testing SUL with current data, merged with current SUL so that we can use same logins on all sites
13:19 petan|work: updated svn
12:44 johnduhart: Ran update.php on metawiki
12:32 johnduhart: Created enwikibooks http://en.wikibooks.deployment.wmflabs.org/wiki/Main_Page
08:12 johnduhart: Starting import of metawiki
07:37 johnduhart: This only affects my new configuration though
07:36 johnduhart: WARNING: some how new users are showing up on some feed outside of labs, and was picked up on a monitoring robot. wtf.
07:15 johnduhart: Importing simplewikibooks
07:10 johnduhart: SiteMatrix is now working http://meta.wikimedia.deployment.wmflabs.org/wiki/Special:SiteMatrix
07:09 johnduhart: Created simple wikibooks http://simple.wikibooks.deployment.wmflabs.org/
07:09 johnduhart: Adding wikibooks configuration
06:31 johnduhart: Installing git on deployment-test
06:23 johnduhart: Created simplewiki http://simple.wikipedia.deployment.wmflabs.org/wiki/Main_Page
05:50 johnduhart: Downloaded metawiki dump ont dbdump, extracting now. poor labs.
05:36 johnduhart: Recreated hwiki, had a bad JohnTest account. Central auth now works fully
05:15 johnduhart: Running update.php on hiwiki
05:14 johnduhart: Importing prefstats table to hiwiki
04:41 johnduhart: Centralauth is working :)
04:28 johnduhart: Imported db schema to centralauth
04:25 johnduhart: Created centralauth database centralauth
04:18 johnduhart: Created metawiki http://meta.wikimedia.deployment.wmflabs.org/
04:18 johnduhart: http://meta.wikimedia.deployment.wmflabs.org/
04:09 johnduhart: Created meta docroot and added favicons to meta and wp
04:08 johnduhart: Adding remnant.conf to httpd.conf
04:07 johnduhart: Adding metawiki apache config to /usr/local/apache/conf/remnant.conf
04:03 johnduhart: Lot of configuration done, much tuning to come. Missing.php now works http://nope.wikipedia.deployment.wmflabs.org/wiki/Main_Page
01:24 johnduhart: Installed mysql-client on deployment-test

January 8

23:45 johnduhart: Installed php5-cli on deployment-test
23:21 johnduhart: Enabled rewrite rules on deployment-test
20:39 petan: created new group for /mnt/www
19:33 petan: clearing cache
18:47 hexmode: set up relative paths for config files
16:45 petan: moved memcached to deployment-nfs-memc and update global
14:04 petan: created temporary memcached instance for data
12:17 petan: restarted apache to fix some problem with config
00:11 hexmode: set config for timeline
00:11 hexmode: "apt-get install ploticus ttf-freefont" for timeline
00:04 jeremyb: booting memcached fixed it
00:02 jeremyb: booting memcache in case that fixes interwiki
00:02 jeremyb: [auth] added bugzilla to interwiki.sql and ran it, doesn't seem to be working

January 7

23:55 hexmode: Fix path for PoolCounter: PoolCounter.php -> PoolCounterClient.php
23:52 hexmode: Fix path for OAI: OAI.php -> OAIRepo.php
23:31 hexmode: set up git for /var/www/global
23:08 petan: disabled LQT on other wikis
23:07 petan: changed configuration of ajax
23:07 petan: fixed InitialiseSetting
22:42 petan: updated all wikis to latest head
22:36 petan: disabled LQT because it's broken
22:34 petan: ran update on auth db
22:10 petan: removed puppetized memcached, because its configuration suck

January 5

20:26 petan: importing MW ns full history to en_wikipedia
16:48 petan|w: created instance for dumps, current instance is overloaded
15:53 petan|w: import now running using mwimport

January 4

20:27 petan: opened port 80 for wide net
20:26 petan: registered deployment.wmflabs.org
20:24 petan: allocated ip208.80.153.215
20:22 mutante: raised floating IP quota to 1
16:10 petan|work: create instances for apache and mysql
16:08 petan|work: configured firewall for webserver
16:06 mutante: added members MarkAHershberger & Petrb - added them to sysadmin and netadmin roles
16:02 mutante: added new project deployment-prep for hexmode and petan
16:02 mutante: added new project deployment-prep for hexmode and petan

@@ Line 1: / Line 1: @@
+=== May 3 ===
+* 04:48 jeremyb: [deployment-dbdump] (that was to address complaints about beta simplewiki appearing in #cvn-simplewikis on freenode)
+* 04:47 jeremyb: [deployment-dbdump] changed all refs to IPs of prod hosts nfs-home and ekrem to be deployment-feed instead. and commited that to the local repo. (again not pushed anywhere yet)
+* 04:44 jeremyb: [deployment-dbdump] did a checkpoint `git commit -a` on deploymentprep-conf (/usr/local/apache/common) (locally not pushed anywhere) because there were lots of changes on disk but not in the repo. but didn't add any new files to the repo. (so there's still stuff reported uncommited by `git status`)
 === May 2 ===
 * 15:14 labs-logs-bottie: petrb: making syslog on apaches be /data/project/apaches_log