Help:Toolforge/Troubleshooting

From Wikitech
Jump to navigation Jump to search

Overview

This page contains information about common issues you may encounter while working with Toolforge.

Troubleshooting

Permission denied when running a script

Make sure the script is running from your Tool Account, rather than your user account.

User preferences can't be accessed in the replicas

User preferences are considered private information and can't be accessed in the replicas.

Error message: No such tool

You receive the following error message after creating a new Tool Account:

$ become <TOOL NAME>
become: no such tool '<TOOL NAME>'
  • Wait a few minutes for the Tool Account creation to complete.
  • Check that the spelling of the tool name is correct.

Error message: You are not a member of the group tools

You receive the following error after joining a Tool Account:

$ become <TOOL NAME>
You are not a member of the group tools.<TOOL NAME>.
Any existing member of the tool's group can add you to that.

If you are already logged in via SSH when you created the a new tool, log out, and log in again to activate your new permissions.

Error message: Trying to ssh into login.toolforge.org: getting a "x@login.toolforge.org: Permission denied (publickey,hostbased)".

If you are running ssh from the command line, make sure that the command is ssh username@login.toolforge.org, where "username" is your username (and not the name of the tool you're attempting to log into).

Error message: Warning: the ECDSA host key for 'login.toolforge.org' differs from the key for the IP address '208.80.155.130'

This means the key has been updated. Try refreshing it with :ssh-keygen -R <ip>

public_html does not work in home directory

Users cannot have a public_html folder to themselves. The only web-accessible directories are under /data/project/<toolname>/public_html/.

To have a URL such as http://<username>.toolforge.org/, you must create a tool called <username>, which will create a folder called /data/project/<username>/public_html/.

Nobody--except you, the user--will *ever* be given access to your home or its files. Allowing public services to run from a home directory means that their management could not be shared or taken over if they end up abandoned, defeating the purpose.

Can't detect Cloud VPS or the project

Every Cloud VPS instance contains the file: /etc/wmcs-project. The contents of the of the file will indicate whether the project is "tools" for the Toolforge, or "toolsbeta" for the experimental Toolforge.

Tool requires a package that is not currently installed in Toolforge

You might not be the only one missing that package. Please submit a ticket in Phabricator and ask the admins to install it project-wide. If they have reasons not to do so you can always install software locally / just for yourself.

Note that for python packages, we recommend that you use a virtualenv local to your tool.

Only request a global package if it is difficult to install via pip (numpy, scipy, etc). For example to install the python package internetarchive run

virtualenv ~/env
source $HOME/env/bin/activate
echo "source $HOME/env/bin/activate" >> .bashrc
pip install internetarchive

See also Help:Toolforge/Python application stub on how to get a sample flask python application installed onto the tools-project.

Installing Python dependencies fails with build errors

When trying to install Python packages (in a venv, see above), you might get a lengthy build error with messages like "pkg-config is not installed", "Edit setup.cfg to change the build options", "You may need to install the development package", or "The following required packages can not be built". This might especially happen when pip indicates it downloaded a file like libraryname-version.tar.gz instead of libraryname-version-LotsOfOtherStuff.whl. Installing wheel may help:

source $HOME/env/bin/activate
pip install wheel

If you still get the same error, and pip is still not downloading a .whl file, check that you are using a recent version of the library. Old versions may not have a wheel available for download, especially not for newer Python versions. For example, if your requirements.txt contains a line like matplotlib ~= 2.0, check matplotlib on PyPI and consider upgrading to the newest version. (This might require some changes to your code, to make it work with the newer library version.)

Slow connection

Your connection may be slower depending on your location.

If you are connecting from Europe, try using Mosh. To connect, use mosh -a login.toolforge.org (-a to force predictive echo) (instead of ssh login.tooforge.org)

No "instances"

Developers working in Toolforge do not have to create or set up virtual machines (i.e., Cloud VPS "instances"), because the Toolforge project admins create and manage them. The term may appear in global documentation on Wikitech but is not applicable to Toolforge.

Bastion is slow

When your connection is fine but the Toolforge bastion (login.toolforge.org, tools-sgebastion-07) is slow or unresponsive, it is likely due to NFS lag. All users of the Toolforge bastion share the same limited connection to NFS. Tasks such as downloading large files or doing processing directly on the bastion host create a high volume of traffic that saturates the connection to NFS. Interactive tools like your shell, cd, and ls must then wait longer to send and receive their own traffic.

To prevent NFS lag, do not run any tasks that require heavy disk IO directly from the bastion. Instead, run them from the Grid or inside a webservice shell. Interactive jobs that require disk IO can also be carried out on the development bastion (dev.toolforge.org), but they may still impact others there.

Toolforge administrators will kill processes running from the bastion that are causing excessive NFS lag.

Report issues with Toolforge

Phabricator is used for issue and work tracking across Wikimedia technical projects.

All issues in Phabricator are available for the public to view. However, to use Phabricator to report issues, you must create a developer account.

When reporting issues, remember

  • Phabricator is relies on volunteers. It may take some time before your issue is addressed.
  • Make an effort to define the problem thoroughly and specifically.
    • Where did you encounter the problem?
    • What isn't working?
    • How do you think it should be working.

Reporting an issue with a specific Tool

1) Find maintainers to notify and a Phabricator project to associate.

As an example, you can find details for the admin Tool (which powers the default page for admin.toolforge.org) at the toolsadmin page for it. Any tool should provide information at a similar URL: ''https://toolsadmin.wikimedia.org/tools/id/<tool name>''

2) Create a task in Phabricator.

There are general Wikimedia documents on the needed instructions and recommended content for reporting issues.

This can use this template to populate a task form.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support
Stay aware of critical changes and plans
Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)