Portal:Toolforge/Changelog

In this page you will find a list of smaller user-facing updates to Toolforge services, new small features and similar news. Major changes are announced at News and/or the cloud-announce@ mailing list.

2025

2025-11-13: Jobs with multiple replicas will spread in different nodes of the cluster if possible (phab:T408707, thanks User:DamianZaremba!)
2025-11-11: Jobs-api and jobs-cli now support fully qualified image names for buildservices (phab:T408574 ex. toolforge jobs run --image tools-harbor.wmcloud.org/tool-sample-complex-app/cronjob:latest --command...., thanks User:DamianZaremba!)
2025-11-03: Now ennvars-cli will allow extracting the raw value of a secret using --raw for reusage in scripts (phab:T405024, thanks User:DamianZaremba!)
2025-10-30: Jobs-cli pulls logs now from the logs-api directly instead of going to the jobs-api.
2025-10-29: Fixed an issue with jobs-api that was flagging buildpacks by default as mounting NFS, this had been introduces a few days ago, so some jobs might fail restarting if they inadvertently depended on nfs being mounted
2025-10-27: Builds-api now returns the sha of the built image too, so you can compare images even if they have the same tag (thanks User:DamianZaremba!); Jobs-api now reloads gracefully continuous jobs when restarting (also thanks User:DamianZaremba!)
2025-10-14: Components-api will now return, and components-cli show, the configuration used for a given deploy when showing a deployment with toolforge components deployment show. (phab:T400064); Components-api will not return now unset parameters for deployments and tool configuration (phab:T402572); Components-api now supports specifying the port_protocol for continuous components (phab:T401994)
2025-09-25: Pod cpu request and limits have been changed (request reduced, limit increased), that should allow better usage of our infra and reduce the current over reservation of resources (phab:T404726); Logs now are fetched every second, so short-lived pods should not be missed anymore, and the delay from starting a job to getting logs should be reduced (phab:T404176); Jobs-api now considers the three types of jobs different, and uses different models, no functional changes though API definitions are now clearer, and opens the way for further improvements(phab:T390136)
2025-09-18: Components reusing other component build, now get rerun if the component they reuse the build from gets rebuilt (phab:T403285, thanks User:DamianZaremba!)
2025-09-12: The bastion servers were upgraded to Debian 13.
2025-09-03: When fetching jobs from the jobs-api, now you can use a query parameter include_unset to include, or not, the job fields that were not sent when the job was created, and set to the default values (phab:T402569).
2025-08-28: Added support for reusing builds between components with `reuse_from` config key, see Help:Toolforge/Deploy your tool#Reusing builds between components (phab:T401893 thanks User:DamianZaremba!)
2025-08-12: From now on, jobs returned by the jobs-api will not have the launcher prefix, though it will still automatically add it for buildservice-based images, users don't need to specify it.; toolforge envvars list now does not show the variables values by default, you have to pass --show-values to show them (phab:T363544).; Added Prebuilt images based on Debian 13 "Trixie".
2025-08-11: Components-api now allows you to set source_url on your tool configuration to pull the configuration before every deployment (see Help:Toolforge/Deploy your tool#Fetching your tool configuration from a URL on deployment).; Components-api now accepts the use_latest_versions option in the build section (see schema).; Components-cli can use stdin when creating the tool configuration, that enables things like curl https://url.to/my/config.yaml | toolforge components config create (phab:T398424).; Fixed autocomplete of the toolforge components deployment cancel subcommand.
2025-07-31: Changed toolforge jobs logs to query logs from the new log storage system.
2025-07-14: Added gitlab scripts to deploy by just including and setting the auth from any gitlab tool.
2025-07-10: Added support to retrieve a deployment by id using the deploy-token so CI systems can loop and wait for the deployment to finish (see the help page).
2025-07-03: Added support for cancelling an ongoing deployment (T395039, Help:Toolforge/Deploy your tool#Cancel an ongoing deployment), and added auto-completion for the toolforge components config create subcommand (T395077).
2025-07-02: Added support for scheduled components to the tool configuration (deployment/push-to-deploy).
2025-06-30: Announced the new push-to-deploy features open beta on cloud-admin list.
2025-06-12: The Toolforge admin console now uses an interface exposed by Wikimedia IDM for username validation. This fixes some bugs with trying to create developer or tool accounts with names that match existing Wikimedia SUL accounts.
2025-06-04: The sql utility now supports connecting to extension database clusters with the --extension flag.
2025-05-22: Toolforge-hosted web services can now be accessed over IPv6.
2025-05-14: When restarting a continuous job, now it will stop the old one completely before attempting to run the new instance, avoiding both of them running at the same time, but causing some downtime (T375366)
2025-05-12: Added a new flag to the builds-cli to use the newest runtime versions toolforge build start --use-latest-versions [your-url-here] (T380127), this currently includes ubuntu 24 runtime, golang 1.24, poetry support, and other fixes
2025-04-23: Now when creating a one-off job, if there's one that already exists with the same name but it's completed, it will just overwrite it instead of failing (T352989)
2025-04-15: Fixed a cron parsing bug for old jobs (T391786)
2025-04-11: jobs-api now shows also the container when showing the logs (T388274); jobs-api now streams logs from all the containers in the job if it has more than one, not only the first one (T388274)
2025-04-08: Limited the amount of parallel builds to 4 (T388706)
2025-04-07: Builds have now a new pending state before they are picked up by tekton (T389954)
2025-03-11: Now users are not allow to manually add any hostPath when directly creating k8s objects, instead they have to rely on the volume-admissison component to do it for them by setting the toolforge.org/mount-storage label to all (T386921).; Fixed bug when streaming logs from jobs-api (T356503)
2025-02-25: Added --timeout <N> to toolforge jobs run --schedule... to specify a timeout in seconds to stop your job if it's stuck, this relieves a bit the issues of NFS getting stuck, and gives a chance for the next trigger to run in a different k8s worker. (T306391)
2025-01-29: Added --health-check-http to toolforge jobs run --continuous... to specify a http endpoint that will be used to check the health of a continuous job and restart it when unhealthy. (learn more)

2024

2024-11-25: Added three global environment variables with the Elasticsearch, Redis and Toolforge API URLs. (see Help:Toolforge/Envvars and the global envvars definitions)
2024-11-14: Changed toolforge webservice to output warning if there are any unknown keys in the service.template file.
2024-09-20: Added --replicas to continuous jobs to allow for running multiple job instances/processes. (learn more)
2024-08-05: Announced the global availability of the Toolforge API.
2024-07-11: Rolled out toolforge-jobs 16.0.13 that can run inside buildservice containers. This is the first step towards being able to run the clients inside k8s.
2024-06-11: Increased the default quota for Deployments (continuous jobs and webservices) from 3 to 16.
2024-05-30: Added --port to continuous jobs to allow backend services in a single tool (learn more)
2024-04-03: Added a --health-check-script to toolforge jobs run --continuous to specify a script to run that will check the health of a continuous job and restart it when unhealthy. (learn more); Added toolforge jobs dump subcommand to generate a YAML file with the current jobs that can be used toolforge jobs load. (learn more)
2024-03-14: Turned off the Grid Engine. Kubernetes is now the only runtime available.
2024-03-07: Added support for jobs using buildservice images to enable file logging with toolforge jobs run --filelog ...
2024-02-28: Added a TCP health check probe to webservices by default. Added support for configuring an HTTP probe with the --health-check-path option (for example --health-check-path /healthz).
2024-02-13: Change the build service to users when starting a new build without having much quota left (and added some followup options).
2024-01-31: Fixed support for envvars on lighthttp (php7.4, etc.) based k8s images; Improved Apt buildpack dependency resolution and rolled out broken link fixes.
2024-01-11: Restored support for clojure, which was dropped from upstream and got removed it with the latest builder upgrade.
2024-01-05: Updated the builder image to heroku/builder:22 latest, with 2.6.1 nodejs buildpack supporting pnpm install
2024-01-04: Added support for passing envvars through stdin or hidden prompt.

2023

2023-12-19: Added support for Rust tools to the build service.
2023-12-15: Added the toolforge build clean subcommand to allow freeing quota for the build service.
2023-12-13: The --wait argument of the jobs framework now supports specifying a custom timeout.; Added support for dotnet/mono stack for the build service.; Added support to specify build environment variables to the toolforge build command (toolforge build start --envvar MYVAR=myvalue --envvar MYVAR2=myvalue2 https://url/to/my/repo.git).; Created the My first Buildpack static tool tutorial.
2023-12-12: Added a note in the My first Buildpack PHP tool on how to serve the index.php from a subdirectory.
2023-12-08: Added the `toolforge build quota` subcommand to check how much resources you are using (working on a way to release them if needed).
2023-12-04: Added support for following build logs (as tail -f would) with the build service, which is enabled by default when starting a build.
2023-11-17: Added a Pywikibot image for running Pywikibot scripts.
2023-10-19: Added support for canceling jobs in the build service.
2023-10-12: Added support running build service images without NFS storage mounted using the jobs framework.
2023-10-11: Created this Toolforge changelog page for small updates! :)
2023-10-05: Added support for supports building static assets with NodeJS while having any other language for the service (Python/Ruby/...) in the build service.

Communication and support

Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:

Discuss and receive general support

Chat in real time in the IRC channel #wikimedia-cloud ^connect or the bridged Telegram group
Discuss via email after you have subscribed to the cloud@ mailing list

Stay aware of critical changes and plans

Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
Read the News wiki page

Track work tasks and report bugs

Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself

Read stories and WMCS blog posts

Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)