User:MichaelSchoenitzer/Workflows and Tips

From Wikitech

Deploying your tool

Workflows, Tips and Tricks how to get your code running on Toolforge.

Uploading

To get your code running, you first have to upload it to the right folder in the home-directory of your tools account.

scp/rsync

scp it the usual way to copy files to a remote host. In the case of Toolforge you have the hitch that the tool has it's own account and you cannot login to this account directly, but need to first login with our account and then use the become script. That means that you can only do uploads with scp with your account, not the tools account. Your account is however part of the group that corresponds to your tool, this allows you to upload to (subfolders of) the tools home directory if you change the user rights to allow changes to the folder by people from the group.

Example:

$ ssh youusername@login.tools.wmflabs.org
$ become yourtoolname
$ mkdir -p www
$ chmod g+w www
$ <C-g>
$ <C-g>
$ scp file youusername@login.tools.wmflabs.org:/data/project/yourtoolname/www/

Make sure to only allow your group write access, not for everybody – the folder should have the permission rwxrwxr-x (775) not 777!

Once you have the permissions set up for scp, you can use rsync to upload or update the whole folder via ssh-access.

lsyncd

lsyncd is a tool, that can monitor your local files and upload them via ssh/rsync whenever you change one of them. This is especially useful if you are working on your tool and want all changes you made to be uploaded directly so you can see/test them. (See also bellow) Since lsyncd is build on top of scp/rsync you need to set up the permissions as above first.

To use it create a config file lsyncd.conf in your local folder:

sync {
    default.rsync,
    source    = "/home/you/Code/greattool/",
    target    = "login.tools.wmflabs.org:/data/project/project/www/python/src/",
    delay     = 10,
        exclude = { '.git', 'configfile.py'},
        excludeFrom = '.gitignore',
    rsync     = {
        backup = true,
        cvs_exclude       =  true,
    }
}

This will upload changes to files in your source directory to the project folder 10 seconds after you change them in your local folder. It will ignore the config files, the git-folder and files in the gitignore file and creates a backup of preexisting destination files (rsync option -b/--backup).

git manual push/pull

Probably the easiest way to get your code to production on Toolforge (if you are familiar with git) it to push it from your code to some remote (for example Github) and pull it on Toolforge.

local$ git push
toolforge$ git clone https://github.com/you/yourtool
toolforge$ git pull

git with hooks

Todo

Uwsgi: Reloading

When deploying changes you want to make sure the changed files are used. Reloading the webservice (webservice --backend=kubernetes xxx restart) works but results in several minutes long downtimes. Instead, you might want to trigger uwsgi to reload the source code. Uwsgi has a few options for that.

Hacky solution from MichaelSchoenitzer (talk) using a fifo

Create the uwsgi config file (uwsgi.ini) if it doesn't exist already and add:

[uwsgi]
master-fifo = /data/project/yourtool/uwsgi.fifo

Writing the character "r" to this fifo will trigger reload of your code – however this only works from the same host as uwsig is running. You can find the name of the "pod" your application is running on with:

$ kubectl describe pod | head -1

Then send the reload trigger:

$ kubectl exec yourtool-987f123456-4prhs -- sh -c "echo r > /data/project/yourtool/uwsgi.fifo"
FIXME: Know a better solution? Please add it here.

Rapid testing for developement

Setting up your accounts rc-files

Reading database credentials