User:Magnus Manske/Migrating from toolserver

From Wikitech

This page describes my personal experiences with porting my tools from the toolserver to labs. I hope it can help others.

Concept

Labs is based on projects, and (usually several) virtual machines (VMs) within each project. For migration from the toolserver, you want to become a member of project "tools". To achieve that:

  1. Create an account on this wiki; you will have both a "user name" and a "shell name". The "shell name" is your unix account.
  2. Create SSH keys and upload your public key here, as described at https://wikitech.wikimedia.org/wiki/Help:SSH
  3. Fill in the Tools Access Request form.
  4. Once your account has been accepted, you can login to tools-login: ssh -A SHELLNAME@tools-login.wmflabs.org.

Create a new tool

  1. Once you are part of project "tools", go to the Tool Labs webpage.
  2. Tools on the shared VMs are called "service groups". Each service group is a tool, and has:
    • a UNIX user "local-TOOL"
    • a UNIX group "local-TOOL"
    • a directory /data/project/TOOL, with subdirectories public_html for HTML/JS/CSS, and cgi-bin for Perl/PHP scripts not sure what would go here
  3. To create a new project, on the Tool Labs webpage (next to "Hosted tools"), click on "create new tool". Enter the TOOL name (without "local-"!). This will:
    • create the tool directories. public_html will be visible under http://tools.wmflabs.org/TOOL, and cgi-bin under http://tools.wmflabs.org/TOOL/cgi-bin
    • add your UNIX user to the UNIX group "local-TOOL"
    • create the UNIX user "local-TOOL", also in that UNIX group. After logging out and in, you can act as that user via ssh with become TOOL, but you can also create and edit files under your "normal" UNIX name
      Note: It seems that the system takes a few minutes until your "normal" UNIX name can write to the tool directories. Give it half an hour after tool creation to be on the safe side, or work as the "local-TOOL" user.
  4. There is help page about accessing Labs Tools with PuTTY and WinSCP. The tools project supports direct ssh login at tools-login.wmflabs.org, no need to connect through the labs bastion.
  5. For a full server/status list, see Nova Resource:Tools.

jargon

The jargon used for the TS and on Labs is slightly different. See below for some hints.

Toolserver Labs
Cluster (e.g. 'the Toolserver Cluster') Project (e.g. 'the Tools Project')
Roots Project Admins
Multi-maintainer Project Service group

qsub et al

Tool Labs also uses SGE for job scheduling. There are a few things to take into account when moving a tool:

  • qcronsub is not available - use qsub instead (same syntax, but doesn't stop multiple jobs) or jsub
  • Tool Labs uses a different memory limit system - instead of virtual_free (which does not count shared libraries), it uses h_vmem. For instance, instead of -l virtual_free=50M, use -l h_vmem=256M.
  • There are no Solaris servers!

PHP

PHP scripts (other languages probably as well) go into public_html, not cgi-bin. Also, the "local-TOOL" user has to be the owner of the file and the subdirectories within public_html, otherwise you'll get a 500 error!

Slow connections

If you're in Europe, and used to the Toolserver, you might notice the somewhat higher ping times (170-200 ms compared to 20 ms). To mitigate this, try using mosh (for windows, follow these somewhat more complicated instructions).

To connect, use 

mosh -a tools-login.wmflabs.org

(-a to force predictive echo) instead of the normal 

ssh tools-login.wmflabs.org

Connecting to different servers

If you want to ssh to specific bot instances other than tools-login.wmflabs.org, it is helpful to create a new SSH key: 

$ ssh-keygen
$ cat ~/.ssh/id_rsa.pub
ssh-rsa .... user@host

Copy the ssh-rsa ... user@host line to your authorized keys in the labs console.

Database access

Replicated databases

The sql command works for both you and the "tool user" on tools-login: 

sql enwiki_p

For your bots and web tools, read the user name and password for the MySQL access from the 

/data/project/TOOL/replica.my.cnf

file. The database name is the usual format 

enwiki_p

and the server name is build by omitting the "_p" and appending ".labsdb" : 

enwiki.labsdb

Create new databases

You can create databases on any of the replicas and on tools-db, as long as the database name starts with the username (that is the database username, not the tool user name!) from replica.my.cnf, followed by two underscores: 

USERNAME__DATABASENAME

Each tool also has its own database on server 

tools-db

The name of the database is the name of the tool with anything but letters and digits removed ("super-tool" → "supertool"). You cannot access this database with the replica credentials, but must use the credentials in ~/.my.cnf. New tools don't get the .my.cnf file, but the replica credentials will work.

See also documentation here.

GIT

WMF Labs does not offer an easy-to-use code repository. However, you can use a local git to version and keep backups, or BitBucket for unlimited git repositories (up to 5 users per git).

Note that in order to be able to push to BitBucket or GitHub from the Tool Labs server, the shell you use to ssh to tools-login needs to have both of your ssh keys available (if they're different). And, regardless of whether they're different, you need to have key forwarding enabled for your ssh session (this is done by passing -A to ssh, like ssh -A tools-login.wmflabs.org

BitBucket

  1. Create an account on https://bitbucket.org, e.g. MYBITBUCKETUSER
  2. Create new repository on bitbucket, e.g. MYTOOL
  3. ssh into tools-login, go to /data/project/MYTOOL
  4. Run git init
  5. Run git remote add origin https://MYBITBUCKETUSER@bitbucket.org/MYBITBUCKETUSER/MYTOOL.git
  6. Add files to git, e.g. git add public_html
  7. Run git commit -m "Initial commit"
  8. Run git push -u origin --all

GitHub

  1. Create an account on https://github.com, e.g. MYGITHUBUSER (e.g. this should be related to you, e.g. your Wikimedia user name, not the Tool)
  2. Create new repository on GitHub, e.g. MYTOOL
  3. ssh into tools-login, go to /data/project/MYTOOL
  4. Run git init
  5. Add files to git, e.g. git add public_html
  6. Run git remote add origin git@github.com:MYGITHUBUSER/MYTOOL.git
  7. Run git commit -m "Initial commit"
  8. Run git push -u origin master

Local git

A local repository will keep versioned backups of your code, but if you accidentally delete the whole directory, this will go too!

  1. Run git init
  2. Add files to git, e.g. git add public_html
  3. Run git commit -m "Initial commit"

Unsolved issues

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=User:Magnus_Manske/Migrating_from_toolserver&oldid=1820725"