News/2025 New Toolforge log storage

Toolforge

• Cloud Services overview
• Toolforge user docs
• Toolforge changelog

Get started

• Quickstart: set up and get access
• How Toolforge works
• Rules you must follow
• Tutorials

Build and run tools

• Navigate tool accounts and files
• Build container images for tools
• Run a web service
• Schedule and manage jobs
• Manage tool runtime configuration (envvars)
• Deploy your tool on every push (beta)
• Language-specific details:
  • Python
  • Pywikibot
  • Node.js
  • PHP
  • ...more languages/frameworks
• Use Redis for caching
• Index content with Elasticsearch

Access shared storage and databases

• Access shared storage and public wiki dumps
• Access the Wiki Replicas databases
• Access replica search indices
• Manage tool databases
• Sending and receiving email as tools

Share and maintain tools

• Set up version control and code review
• Develop successful tools
• Find and share tools on Toolhub
• Delete a tool

Get help

• How and where to get help
• Troubleshooting
• Contribute to Toolforge

Useful links

• Toolforge admin docs
• List of tools
• Toolforge Admin Console (toolsadmin)
• Toolforge API

[]

This page is currently a draft.
Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page.

This page contains information about the new log storage system being introduced for Toolforge jobs.

What is changing?

The log storage system for Toolforge jobs is being migrated to a Grafana Loki deployment, instead of relying on the internal log storage handled by the Kubernetes (the container orchestrator we use). This change means that the toolforge jobs logs command is now able to query logs not just from currently running containers.

In particular, toolforge jobs logs is gaining the following features:

• Logs will now be visible from containers that are no longer running. Logs are no longer lost when, for example, a job is restarted or a cron job launch completes
• Logs can be queried from jobs that have since been deleted (phab:T400913)
• Logs can be queried for a specific time frame (phab:T400917)

The log viewing functionality of the webservice command is unaffected by these changes.

Timeline

• Done 2025-07-31: toolforge jobs logs is changed to query logs from Loki.
• In progress New features are added to toolforge jobs logs to take advantage of Loki's functionality.
• TBD: File logging will be disabled by default for new jobs.
• TBD: File logging is removed as a built-in option. Current jobs using file logging will be migrated to use internal logging instead.

Why are we doing this?

This change is being made to improve the reliability of Toolforge by reducing usage of the shared storage (NFS) server.

See also

• Help:Toolforge/Running jobs § Job logs

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

• Chat in real time in the IRC channel #wikimedia-cloud ^connect or the bridged Telegram group
• Discuss via email after you have subscribed to the cloud@ mailing list

Stay aware of critical changes and plans

• Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
• Read the News wiki page

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)