This page contains information about common issues you may encounter while working with Toolforge.
Permission denied when running a script
Make sure the script is running from your Tool Account, rather than your user account.
User preferences can't be accessed in the replicas
User preferences are considered private information and can't be accessed in the replicas.
Error message: libgcc_s.so.1 must be installed for pthread_cancel to work?
If you encounter this while trying to run a job on the grid engine, the job needs more memory.
See Allocating additional memory for more information.
Error message: No such tool
You receive the following error message after creating a new Tool Account:
$ become <TOOL NAME> become: no such tool '<TOOL NAME>'
- Wait a few minutes for the Tool Account creation to complete.
- Check that the spelling of the tool name is correct.
Error message: You are not a member of the group tools
You receive the following error after joining a Tool Account:
$ become <TOOL NAME> You are not a member of the group tools.<TOOL NAME>. Any existing member of the tool's group can add you to that.
If you are already logged in via SSH when you created the a new tool, log out, and log in again to activate your new permissions.
Error message: Trying to ssh into login.toolforge.org: getting a "email@example.com: Permission denied (publickey,hostbased)".
If you are running
ssh from the command line, make sure that the command is
ssh firstname.lastname@example.org, where "username" is your username (and not the name of the tool you're attempting to log into).
Error message: Warning: the ECDSA host key for 'login.toolforge.org' differs from the key for the IP address '126.96.36.199'
This means the key has been updated. Try refreshing it with :
ssh-keygen -R <ip>
public_html does not work in home directory
Users cannot have a public_html folder to themselves. The only web-accessible directories are under
To have a URL such as
http://<username>.toolforge.org/, you must create a tool called
<username>, which will create a folder called /data/project/<username>/public_html/.
Nobody--except you, the user--will *ever* be given access to your home or its files. Allowing public services to run from a home directory means that their management could not be shared or taken over if they end up abandoned, defeating the purpose.
Can't detect Cloud VPS or the project
Every Cloud VPS instance contains the file:
/etc/wmcs-project. The contents of the of the file will indicate whether the project is "tools" for the Toolforge, or "toolsbeta" for the experimental Toolforge.
Tool requires a package that is not currently installed in Toolforge
You might not be the only one missing that package. Please submit a ticket in Phabricator and ask the admins to install it project-wide. If they have reasons not to do so you can always install software locally / just for yourself.
Note that for python packages, we recommend that you use a virtualenv local to your tool.
Only request a global package if it is difficult to install via pip (numpy, scipy, etc). For example to install the python package internetarchive run
virtualenv ~/env source $HOME/env/bin/activate echo "source $HOME/env/bin/activate" >> .bashrc pip install internetarchive
See also Help:Toolforge/Python application stub on how to get a sample flask python application installed onto the tools-project.
Installing Python dependencies fails with build errors
When trying to install Python packages (in a venv, see above), you might get a lengthy build error with messages like "pkg-config is not installed", "Edit setup.cfg to change the build options", "You may need to install the development package", or "The following required packages can not be built". This might especially happen when pip indicates it downloaded a file like
libraryname-version.tar.gz instead of
libraryname-version-LotsOfOtherStuff.whl. Installing wheel may help:
source $HOME/env/bin/activate pip install wheel
If you still get the same error, and pip is still not downloading a
.whl file, check that you are using a recent version of the library. Old versions may not have a wheel available for download, especially not for newer Python versions. For example, if your
requirements.txt contains a line like
matplotlib ~= 2.0, check matplotlib on PyPI and consider upgrading to the newest version. (This might require some changes to your code, to make it work with the newer library version.)
Your connection may be slower depending on your location.
If you are connecting from Europe, try using Mosh. To connect, use
mosh -a login.toolforge.org (-a to force predictive echo)
Developers working in Toolforge do not have to create or set up virtual machines (i.e., Cloud VPS "instances"), because the Toolforge project admins create and manage them. The term may appear in global documentation on Wikitech but is not applicable to Toolforge.
Bastion is slow
When your connection is fine but the Toolforge bastion (login.toolforge.org, tools-sgebastion-07) is slow or unresponsive, it is likely due to NFS lag. All users of the Toolforge bastion share the same limited connection to NFS. Tasks such as downloading large files or doing processing directly on the bastion host create a high volume of traffic that saturates the connection to NFS. Interactive tools like your shell,
ls must then wait longer to send and receive their own traffic.
To prevent NFS lag, do not run any tasks that require heavy disk IO directly from the bastion. Instead, run them from the Grid or inside a webservice shell. Interactive jobs that require disk IO can also be carried out on the development bastion (dev.toolforge.org), but they may still impact others there.
Toolforge administrators will kill processes running from the bastion that are causing excessive NFS lag.
Report issues with Toolforge
Phabricator is used for issue and work tracking across Wikimedia technical projects.
All issues in Phabricator are available for the public to view. However, to use Phabricator to report issues, you must create a developer account.
When reporting issues, remember
- Phabricator is relies on volunteers. It may take some time before your issue is addressed.
- Make an effort to define the problem thoroughly and specifically.
- Where did you encounter the problem?
- What isn't working?
- How do you think it should be working.
Reporting an issue with a specific Tool
1) Find maintainers to notify and a Phabricator project to associate.
As an example, you can find details for the admin Tool (which powers the default page for admin.toolforge.org) at the toolsadmin page for it. Any tool should provide information at a similar URL: ''https://toolsadmin.wikimedia.org/tools/id/<tool name>''
2) Create a task in Phabricator.
There are general Wikimedia documents on the needed instructions and recommended content for reporting issues.
This can use this template to populate a task form.
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect, the bridged Telegram group, or the bridged Mattermost channel
- Discuss via email after you subscribed to the cloud@ mailing list