News/Toolforge Grid Engine deprecation
This page contains information about the deprecation and removal of the Toolforge Grid Engine platform.
What is changing?
While no specific date has been set, the Grid Engine cluster will be decommissioned once all tools have been migrated off the Grid.
The Toolforge admins are working with maintainers, collecting blocking issues, and assisting in moving tools off the Grid.
This work is being tracked on the Phabricator workboard
Timeline
This is a rough timeline and could be taken as simple date placeholders. We hope that future edits to this section may introduce more precision.
- FY21/22 Q2 (Oct-Dec 2021): finish work & release the Toolforge Jobs Framework. Continue working on Toolforge buildpacks. Migrate Son of Grid Engine to Debian Buster.
- FY22/23 Q2 (Oct-Dec 2022): Ask community to begin migrating tools. Collect blocking issues.
- FY22/23 Q3 (Jan-Mar 2023): Add features to support identified blocking issues. Explore k8s a service as potential migration path phab:T328712 ? Tool migrations continue.
- FY22/23 Q4 (Apr-Jun 2023): Toolforge buildpacks beta. See phab:T267374 Tool migrations continue.
- FY23/24 Q1 (Jul-Sep 2023): Toolforge buildpacks multipack support work. See phab:T325799 . Tool migrations continue.
- FY23/24 Q2 (Oct-Dec 2023): Most tools are able to be migrated?
What should I do?
You have a couple of options:
- migrate your Toolforge tool to Toolforge Kubernetes.
- migrating web services
- migrating jobs
- simply delete your tool, in case it is not used anymore.
Use case continuity
The following table tracks use case continuity.
| Feature | Grid Engine | Kubernetes | Comment |
|---|---|---|---|
| job scheduling | jsub or jstart | Toolforge jobs One off jobs or continuous jobs |
Example:
From GridEngine 5 * * * * jsub -once -N name-of-tool $HOME/user/bot.php >/dev/null 2>&1
To Kubernetes $ toolforge-jobs run name-of-tool --command ./user/bot.php --image php7.4 --schedule "5 * * * *"
|
| web services | webservice | specify an image and 'kubernetes' as the backend | Example:
From GridEngine $ webservice --backend=gridengine start
To Kubernetes $ webservice stop
$ webservice --backend=kubernetes php7.4 start
|
| Multi-language tools | Native | Toolforge buildpacks | Some single language tools will need updated or new images (like dotnet) |
Why are we doing this?
As outlined in our series of blog posts, Toolforge is powered by two different backend engines, Kubernetes and Grid Engine. These two backends have traditionally offered different features for tool developers. But as time moves forward we’ve learnt that Kubernetes is the future.
See more for a detailed explanation.
Solutions to common problems
Rebuild virtualenv for Python users / python3: not found
Python virtual environments ("venvs") are tied to the underlying system where they are running. Because of that, you will need to delete and re-create your virtual environments using these instructions.
Tools needing multiple language runtimes
Work is underway with the buildpack project.
Mono container
Using mono? See discussion on a Mono specific container phab:T311466
Requires a system library or tool to be present
If you believe this depedency makes sense to have in the base container, file a ticket. For example, mysqldump phab:T254636, resolved.
For other needs, note that buildpack project will support more nuanced and custom container setups.
Pywikibot image
Running pywikibot in the suggest setup Help:Toolforge/Pywikibot#Using_the_shared_Pywikibot_files_(recommended_setup) ?
See phab:T249787 for discussion on a dedicated image for this use case
Delete a tool
Some tools were experiments that are done, others were made obsolete by other tools, some are just things that the original maintainer is tired of caring for. Maintainers can mark their tools for deletion using the "Disable tool" button on the tool's detail page on https://toolsadmin.wikimedia.org/. Disabling a tool will immediately stop any running jobs including webservices and prevent maintainers from logging in as the tool. Disabled tools are archived and deleted after 40 days. Disabled tools can be re-enabled at any time prior to being archived and deleted.
See also
- Wikimedia Techblog: Toolforge and Grid Engine
- Wikimedia Techblog: Toolforge GridEngine Debian 10 Buster migration
- Wikimedia Techblog: Toolforge Jobs Framework
- List of tools still running on the grid engine
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia Movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect, the bridged Telegram group, or the bridged Mattermost channel
- Discuss via email after you subscribed to the cloud@ mailing list