User:BryanDavis/Scap3 in a Labs project

From Wikitech
Jump to: navigation, search

Setting up a deploy server to use scap3 in a Labs project

  • Create a m1.small instance
  • Add role and some dummy hiera config values that are needed by ::role::deployment_server to Hiera:$PROJECT/host/$HOST
---
classes:
    - role::deployment_server
keyholder::require_encrypted_keys: nope
mediawiki_memcached_servers:
    - 127.0.0.1:11211:1
redis::shards:
  jobqueue: &id001
    eqiad:
      shard01:
        host: 127.0.0.1
        port: 6379
  sessions: *id001
scap::dsh::groups:
  mediawiki-installation:
    hosts:
    - 127.0.0.1
  • Add some project wide hiera config values to set the scap and scap3 master server in Hiera:$PROJECT:
    ---
    scap::deployment_server: <FQDN of project deploy server>
    scap::wmflabs_master: <FQDN of project deploy server>
    
  • Force another puppet run and grab a cup of tea. This is going to take a while.

Adding a scap3 project

scap::keyholder_agents:
    deploy-service:
        trusted_groups:
            - wikidev

scap::sources:
    striker/deploy:
        repository: labs/striker/deploy

Syncing with the cluster

  • Accept the ssh host keys of all of the target nodes as the user you will be deploying as (e.g. bd808 if you happen to be BryanDavis)
  • Arm Keyholder on your deploy server.
  • Deploy stuff!
    $ cd $MY_DEPLOY_DIR  # (e.g /srv/deployment/striker/deploy)
    $ scap deploy
    21:14:10 Started Deploy: striker/deploy
    Entering 'public_html/staticfiles'
    Entering 'striker'
    Entering 'wheels'
    21:14:10
    == DEFAULT ==
    :* striker-uwsgi01.striker.eqiad.wmflabs
    striker/deploy: fetch stage(s): 100% (ok: 1; fail: 0; left: 0)
    striker/deploy: config_deploy stage(s): 100% (ok: 1; fail: 0; left: 0)
    striker/deploy: promote and restart_service stage(s):   0% (ok: 0; fail: 0; left: 1)
    striker/deploy: promote and restart_service stage(s): 100% (ok: 1; fail: 0; left: 0)
    striker/deploy: promote and restart_service stage(s): 100% (ok: 1; fail: 0; left: 0)
    21:14:13 Finished Deploy: striker/deploy (duration: 00m 02s)
    $ scap deploy-log
    -- Opening log file: '/srv/deployment/striker/deploy/scap/log/scap-sync-2016-07-28-0007.log'
    21:14:10 [striker-deploy03] Started Deploy: striker/deploy
    21:14:10 [striker-deploy03]
    == DEFAULT ==
    :* striker-uwsgi01.striker.eqiad.wmflabs
    21:14:11 [striker-uwsgi01.striker.eqiad.wmflabs] Revision directory already exists (use --force to override)
    21:14:12 [striker-uwsgi01.striker.eqiad.wmflabs] Starting new HTTP connection (1): striker-deploy03.striker.eqiad.wmflabs
    21:14:13 [striker-uwsgi01.striker.eqiad.wmflabs] /srv/deployment/striker/deploy-cache/revs/47ea97dc38677aab79bd0c89f1e3ffe7fdc2cbfb is already live (use --force to override)
    21:14:13 [striker-deploy03] Finished Deploy: striker/deploy (duration: 00m 02s)
    

Errors I saw on initial provision

Yes check.svg Done https://gerrit.wikimedia.org/r/#/c/301403/ --
Error: Could not set 'file' on ensure: No such file or directory - /etc/firejail/mediawiki-imagemagick.profile20160726-18098-h7ugbr.lock at 45:/etc/puppet/modules/mediawiki/manifests/init.pp
Error: Could not set 'file' on ensure: No such file or directory - /etc/firejail/mediawiki-imagemagick.profile20160726-18098-h7ugbr.lock at 45:/etc/puppet/modules/mediawiki/manifests/init.pp
Wrapped exception:
No such file or directory - /etc/firejail/mediawiki-imagemagick.profile20160726-18098-h7ugbr.lock
Error: /Stage[main]/Mediawiki/File[/etc/firejail/mediawiki-imagemagick.profile]/ensure: change from absent to file failed: Could not set 'file' on ensure: No such file or directory - /etc/firejail/mediawiki-imagemagick.profile20160726-18098-h7ugbr.lock at 45:/etc/puppet/modules/mediawiki/manifests/init.pp
Yes check.svg Done https://gerrit.wikimedia.org/r/#/c/301404 --
Error: Could not set 'file' on ensure: No such file or directory - /etc/php5/apache2/php.ini20160726-18098-10h241o.lock at 21:/etc/puppet/modules/mediawiki/manifests/php.pp
Error: Could not set 'file' on ensure: No such file or directory - /etc/php5/apache2/php.ini20160726-18098-10h241o.lock at 21:/etc/puppet/modules/mediawiki/manifests/php.pp
Wrapped exception:
No such file or directory - /etc/php5/apache2/php.ini20160726-18098-10h241o.lock
Error: /Stage[main]/Mediawiki::Php/File[/etc/php5/apache2/php.ini]/ensure: change from absent to file failed: Could not set 'file' on ensure: No such file or directory - /etc/php5/apache2/php.ini20160726-18098-10h241o.lock at 21:/etc/puppet/modules/mediawiki/manifests/php.pp
Yes check.svg Done https://gerrit.wikimedia.org/r/#/c/301405 --
Error: Could not set 'file' on ensure: No such file or directory - /home/l10nupdate/.gitconfig20160726-18098-axu2js.lock at 81:/etc/puppet/modules/scap/manifests/l10nupdate.pp
Error: Could not set 'file' on ensure: No such file or directory - /home/l10nupdate/.gitconfig20160726-18098-axu2js.lock at 81:/etc/puppet/modules/scap/manifests/l10nupdate.pp
Wrapped exception:
No such file or directory - /home/l10nupdate/.gitconfig20160726-18098-axu2js.lock
Error: /Stage[main]/Scap::L10nupdate/File[/home/l10nupdate/.gitconfig]/ensure: change from absent to file failed: Could not set 'file' on ensure: No such file or directory - /home/l10nupdate/.gitconfig20160726-18098-axu2js.lock at 81:/etc/puppet/modules/scap/manifests/l10nupdate.pp
Yes check.svg Done https://gerrit.wikimedia.org/r/#/c/301408 --
Notice: /Stage[main]/Mediawiki::Scap/Exec[fetch_mediawiki]/returns: 22:25:55 pull failed: <CalledProcessError> Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/cache/l10n/*.cdb', '--exclude=*.swp', '--no-perms', '--exclude=**/.git', 'deployment-tin.eqiad.wmflabs::common', '/srv/mediawiki']' returned non-zero exit status 10
Notice: /Stage[main]/Mediawiki::Scap/Exec[fetch_mediawiki]/returns:
Error: /usr/bin/scap pull returned 70 instead of one of [0]
Error: /Stage[main]/Mediawiki::Scap/Exec[fetch_mediawiki]/returns: change from notrun to 0 failed: /usr/bin/scap pull returned 70 instead of one of [0]n

What's wrong here

  1. trebuchet and all of its baggage including a bunch of cloned repos
  2. Full MediaWiki scap setup
  3. l10nupdate (via MW scap)
  4. Full MW runtime setup (via MW scap)
  5. Why oh why is hiera('mediawiki::redis_servers::eqiad') embedded in role::memcached?
  6. Several resource ordering issues on initial provision