Help:Toolforge/Troubleshooting

This page contains information about common issues you may encounter while working with Toolforge.

Troubleshooting

Permission denied when running a script

Make sure the script is running from your Tool Account, rather than your user account.

User preferences can't be accessed in the replicas

User preferences are considered private information and can't be accessed in the replicas.

Error message: No such tool

You receive the following error message after creating a new Tool Account:

$ become <TOOL NAME>
become: no such tool '<TOOL NAME>'

Wait a few minutes for the Tool Account creation to complete.
Check that the spelling of the tool name is correct.

Error message: You are not a member of the group tools

You receive the following error after joining a Tool Account:

$ become <TOOL NAME>
You are not a member of the group tools.<TOOL NAME>.
Any existing member of the tool's group can add you to that.

If you are already logged in via SSH when you created the a new tool, log out, and log in again to activate your new permissions.

If you are running ssh from the command line, make sure that the command is ssh username@login.toolforge.org, where "username" is your username (and not the name of the tool you're attempting to log into).

This means that the ssh host key stored on your local machine no longer matches the remote host's key. This usually means that the instance was replaced as part of an operating system upgrade or similar migration. It could however indicate that your ssh session is being compromised by a third party.

Your out of date locally stored host keys can be removed with ssh-keygen -R $IP_ADDRESS and ssh-keygen -R $HOSTNAME. The next time you attempt to connect to the server ssh should prompt you to verify and accept the remote host's key. Help:SSH Fingerprints can and should be used to verify that the host key you are seeing is valid proving that you are not being subjected to a man-in-the-middle attack.

public_html does not work in home directory

Users cannot have a public_html folder to themselves. The only web-accessible directories are under /data/project/<toolname>/public_html/.

To have a URL such as http://<username>.toolforge.org/, you must create a tool called <username>, which will create a folder called /data/project/<username>/public_html/.

Nobody--except you, the user--will *ever* be given access to your home or its files. Allowing public services to run from a home directory means that their management could not be shared or taken over if they end up abandoned, defeating the purpose.

Can't detect Cloud VPS or the project

Every Cloud VPS instance contains the file: /etc/wmcs-project. The contents of the of the file will indicate whether the project is "tools" for the Toolforge, or "toolsbeta" for the experimental Toolforge.

Tool requires a package that is not installed in Toolforge

Custom container images can install arbitrary Apt packages for tools. In addition, many programming languages have tools to install packages for that user only, such as virtual environments for Python.

Installing Python dependencies fails with build errors

When trying to install Python packages (in a venv, see above), you might get a lengthy build error with messages like "pkg-config is not installed", "Edit setup.cfg to change the build options", "You may need to install the development package", or "The following required packages can not be built". This might especially happen when pip indicates it downloaded a file like libraryname-version.tar.gz instead of libraryname-version-LotsOfOtherStuff.whl. Installing wheel may help:

source $HOME/env/bin/activate
pip install wheel

If you still get the same error, and pip is still not downloading a .whl file, check that you are using a recent version of the library. Old versions may not have a wheel available for download, especially not for newer Python versions. For example, if your requirements.txt contains a line like matplotlib ~= 2.0, check matplotlib on PyPI and consider upgrading to the newest version. (This might require some changes to your code, to make it work with the newer library version.)

Slow connection

Your connection may be slower depending on your location.

If you are connecting from Europe, try using Mosh. To connect, use mosh -a login.toolforge.org (-a to force predictive echo) (instead of ssh login.tooforge.org)

No "instances"

Developers working in Toolforge do not have to create or set up virtual machines (i.e., Cloud VPS "instances"), because the Toolforge project admins create and manage them. The term may appear in global documentation on Wikitech but is not applicable to Toolforge.

Bastion is slow

When your connection is fine but the Toolforge bastion (login.toolforge.org, tools-sgebastion-07) is slow or unresponsive, it is likely due to NFS lag. All users of the Toolforge bastion share the same limited connection to NFS. Tasks such as downloading large files or doing processing directly on the bastion host create a high volume of traffic that saturates the connection to NFS. Interactive tools like your shell, cd, and ls must then wait longer to send and receive their own traffic.

To prevent NFS lag, do not run any tasks that require heavy disk IO directly from the bastion. Instead, run them inside a webservice shell. Interactive jobs that require disk IO can also be carried out on the development bastion (dev.toolforge.org), but they may still impact others there.

Toolforge administrators will kill processes running from the bastion that are causing excessive NFS lag.

There is no C/C++ compiler

Since the decomission of Grid Engine, the new bastion servers based on Debian Bookworm no longer have various compiler tools and language runtimes installed. If you need to compile something, please create a Kubernetes shell. Here is the process:

Become a tool user; if you do not have a tool you need to create one.
When using a tool account, run webservice python3.11 shell
Inside this new shell you can use gcc

Report issues with Toolforge

Phabricator is used for issue and work tracking across Wikimedia technical projects.

All issues in Phabricator are available for the public to view. However, to use Phabricator to report issues, you must create a developer account.

When reporting issues, remember

Phabricator is relies on volunteers. It may take some time before your issue is addressed.
Make an effort to define the problem thoroughly and specifically.
- Where did you encounter the problem?
- What isn't working?
- How do you think it should be working.

Reporting an issue with a specific Tool

1) Find maintainers to notify and a Phabricator project to associate.

As an example, you can find details for the admin Tool (which powers the default page for admin.toolforge.org) at the toolsadmin page for it. Any tool should provide information at a similar URL: ''https://toolsadmin.wikimedia.org/tools/id/<tool name>''

2) Create a task in Phabricator.

There are general Wikimedia documents on the needed instructions and recommended content for reporting issues.

This can use this template to populate a task form.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

Chat in real time in the IRC channel #wikimedia-cloud ^connect or the bridged Telegram group
Discuss via email after you have subscribed to the cloud@ mailing list

Stay aware of critical changes and plans

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Read the News wiki page

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)