Help:Toolforge/Managing and sharing files

This guide explains how to store, transfer, share, and manage files in Toolforge. It covers common tasks such as choosing the appropriate storage location, protecting sensitive files with the correct permissions, transferring files to tool accounts, and sharing files with other maintainers.

Use this guide when you need to work with files as a Toolforge tool maintainer.

Before you start

This guide assumes that you:

Have a Toolforge shell account.
Are a maintainer of the tool account you want to work with.
Can connect to Toolforge using SSH.
Have a basic understanding of Linux command-line tools and file permissions.

Some examples use placeholder names such as toolaccount, toolname, and yourshellaccountname. Replace these with the appropriate values for your environment.

Understanding Toolforge storage

Toolforge provides several storage locations for tool code, data, and files. These locations differ in who can access them and how they are intended to be used.

As a tool maintainer, it is important to understand which locations are shared, which are private, and which are publicly accessible. Choosing the appropriate storage location helps ensure that files are available to the right users and services while protecting sensitive information.

Choosing where to store files

Before transferring or creating files in Toolforge, choose a storage location based on who needs access to the files and how they will be used.

Toolforge provides several storage locations with different access controls and intended uses.

Location	Access	Intended use
/data/project/toolname	Shared among tool maintainers	Use this location for tool source code, configuration files, and persistent data that must be available to the tool and its maintainers.
/data/scratch	Shared across Toolforge users	Use this location for temporary files, caches, and other data that can be recreated if needed.
/home	Personal workspace	Use it for shell configuration files, small scripts, and work in progress that is not yet part of a tool.
public_html	Publicly accessible	Store files here only if they are intended to be served by a web service and viewed by others.
/public/dumps	Public read-only asscess	Use this location when your tool needs to read dump data without maintaining a local copy.
/data/project/shared/mediawiki	Read-only checkout of MediaWiki repositories	Use this location for code inspection, searching across repositories, and development tasks that require access to MediaWiki source code.

Storage recommendations

Use /data/project/toolname for files that are required by your tool.
Use /data/scratch only for temporary or reproducible data.
Do not store passwords, API keys, or other sensitive information in shared locations.
Do not store databases, log files, or frequently modified data on shared NFS storage.
Store source code in version control rather than relying on shared storage as the primary copy.

Managing file permissions

Toolforge is a shared environment. Files that you create may be accessible to other users, so sensitive files such as passwords, API keys, or configuration files should have restricted permissions.

To create a new private file that is only readable and writable by its owner:

install -m 600 /dev/null FILE_NAME

Choose file permissions carefully before storing sensitive information.

Transferring files

Toolforge supports several ways to move and work with files, depending on your workflow. Common methods include scp for copying files, sshfs for local editing, and rsync for synchronizing directories.

Using scp

You can transfer files from your local machine to Toolforge using scp.

For example, to make a file available to a tool account:

cp somefile ~tools.toolaccount/

After copying a file into a tool account's directory, the tool account must take ownership of the file before it can use it.

Taking ownership of files

The take command changes ownership of files and directories to the calling tool account.

Before using take, the tool account must already have access to the file through either ownership or group permissions.

Become the tool account:

become toolaccount

Take ownership of the file:

take FILE

Working with files locally using sshfs

If you prefer to edit files using a local IDE such as Visual Studio Code or Eclipse, you can mount your Toolforge directory using sshfs instead of editing files directly from the command line.

Create a local mount point and mount the remote directory:

sshfs yourshellaccountname@login.toolforge.org:/data/project/toolname ~/remote

Unmount the directory when finished:

If a normal unmount fails, you can force an unmount:<syntaxhighlight lang="bash">
fusermount -zu ~/remote

Synchronizing files with rsync

For larger or repeated file transfers, rsync can efficiently synchronize local and Toolforge directories.

A basic example:

rsync --archive yourLocalDirectory 
login.toolforge.org:/home/yourshellaccountname/destinationDirectory

To synchronize files directly into a tool account directory, configure rsync to use the appropriate tool account and destination path.

Common problems accessing transferred files

The tool cannot access transferred files

Check that:

The file has been copied to the correct location.
The tool account owns the file.
File permissions allow the required access.

If necessary, become the tool account and run:

take FILE

Sharing files

Toolforge provides several ways to share files and code with other maintainers. Choose a method based on what you are sharing and how it will be maintained.

Share files using a tool account

Files stored in a tool account's project directory can be accessed by maintainers of that tool. This is the preferred way to share configuration files, generated data, and other files that are required by the tool at runtime.

To make a file available to a tool account:

Copy the file into the tool account directory.
Become the tool account.
Transfer ownership of the file to the tool account using the take command.

For example:

yourshellaccountname@tools-sgebastion-10:~$ cp somefile ~tools.toolaccount/

yourshellaccountname@tools-sgebastion-10:~$ become toolaccount
tools.toolaccount@tools-sgebastion-10:~$ take somefile

Store shared files in locations that are appropriate for their purpose:

Use "/data/project/toolname" for persistent tool data.
Use "/data/scratch" for temporary data that can be recreated.

Share code using packages or version control

For source code and reusable libraries, use version control or package management systems instead of manually copying files between tools.

Version control systems such as Git provide a canonical source for code, make changes easier to track, and simplify collaboration between maintainers.
Package managers can be used to distribute reusable code across multiple tools without duplicating files. This helps reduce maintenance overhead and ensures that multiple tools can use the same version of shared code.

When possible:

Store source code in a version control repository.
Distribute reusable libraries as packages.
Use shared storage for runtime data rather than source code.
Avoid maintaining multiple copies of the same code across different tools.

Common problems

Permission denied

Sometimes the file system permissions on a tool's project directory (/data/project/$TOOL) can become misconfigured and lose group write permission. This can prevent tool maintainers from uploading, modifying, or deleting files.

To check the current permissions, log in to a bastion host and inspect the directory:

$ ssh login.toolforge.org
$ ls -ld /data/project/bd808-test
drwxrwsr-x 7 tools.bd808-test tools.bd808-test 4096 Jul 19 15:07 /data/project/bd808-test/

The drwxrwsr-x section of the output describes the directory permissions:

d – a directory
rwx – permissions for the owning user:
- r – read
- w – write
- x – execute
rws – permissions for the owning group:
- r – read
- w – write
- s – sticky. Sticky implies execute and also attempts to set the same group ownership on all new files and directories created inside the directory.
r-x – permissions for other users:
- r – read
- x – execute
- - – no write permission

A common cause of permission errors is that the group write permission (w) is missing from the tool's directory or one of its files or subdirectories.

Ownership issues

Members of a tool account belong to the tool's owning group. When the correct group permissions are missing, tool maintainers may be unable to upload, modify, or manage files.

To restore group write permissions, log in to a bastion host, become the tool account, and run:

$ ssh login.toolforge.org
$ become $MY_TOOL
$ chmod -R g+w /data/project/$MY_TOOL

This command recursively grants write permission to the tool's group for all files and directories within the tool's project directory.

Transfer problems

Incorrect ownership or file permissions can also prevent files from being transferred or updated successfully.

If transferred files cannot be modified by a tool account, check that:

The files were copied to the correct location.
The tool account belongs to the owning group.
The destination directory has group write permission.
Files and subdirectories have inherited the correct permissions.

If necessary, restore the appropriate group write permissions as described in Ownership issues.

Best practices

Follow these recommendations when managing and sharing files in Toolforge:

Store sensitive files, such as passwords, API keys, and configuration files, with restricted permissions.
Use the least permissive file permissions that allow your tool to function correctly.
Choose the appropriate storage location for your files based on whether they need to be private, shared between tools, or publicly accessible.
Use tool accounts and shared storage to collaborate instead of duplicating files across multiple tools.
Prefer package managers or version control systems, such as Git, for sharing reusable code between projects.
Verify file ownership and permissions after transferring files to avoid access issues.
Use rsync for large or repeated file transfers and synchronization tasks.
Regularly review file permissions, especially after moving or modifying files.