Help:Toolforge/Build Service
The Build Service brings Cloud Native Buildpacks to Toolforge. Buildpacks are a specification as well as a piece of code you can put in a repository. A "buildpack" applies user code to a "stack" via a "builder" during the "lifecycle" to get code onto a "platform", which is also the term used to describe the full system.
The goal of the Build Service is to make the process for deploying code to Toolforge easier and more flexible, by adopting a standard that is documented and maintained by a wider community (Buildpacks are a CNCF-supported initiative).
Roadmap
Features that are already available
- Building and running any application that is compatible with one of the Officially supported Heroku buildpacks (python having been tested thoroughly, the others might need some tweaks for now, see current limitations).
- Compared to the current images, this enables you to use more languages than the ones supported by our custom Container images
- Compared to the current images, Python apps are no longer tied to uWSGI and you can now use modern ASGI-based frameworks like FastAPI
- Compared to the current images, you can use newer versions of the base languages (ex. python 3.11, see Specifying a Python runtime), and benefit when newer versions are added upstream.
- Running that application as a webservice
- Limited storage support (see current limitations)
- Running that application as a job
- Limited storage support (see current limitations)
- Pull locally the same image that is going to run in toolforge
Planned features that are not available yet
- Push to deploy (task T341065, task T334587)
- Building assets as part of the build (ex. React + PHP / vue.js + flask, task T325799)
- Other buildpacks in addition to the Officially supported Heroku buildpacks (task T337066)
- Full streamlined storage support
- See the current limitations section for more details and other changes with the current process
Quickstart
Prerequisites
If you don't have a tool account yet, you need to create or join one. Detailed instructions are available at Help:Toolforge/Quickstart.
Your tool's code must be accessible in a public Git repository, any public Git repository will work (Gerrit, GitLab, ...). You can setup one for free for your tool from the Toolforge admin console.
Procfile
You will need to create a Procfile
to configure which commands to run to start your app. The Toolforge webservice manager uses the web
process type, and for jobs you can use whatever process type you want. However, for now, you must have a web
process type defined.
Example: Python web service
For example, the Procfile
for a Python application using Gunicorn (which needs to be installed via requirements.txt) might be this:
web: gunicorn --bind 0.0.0.0 --workers 4 app:app
migrate: python3 -m app.migrate
The first entry will be the one used for webservices if you start it as a webservice (NOTE: no matter it's name, currently is the first one found).
Otherwise they will be used for jobs, where you can have as many entries as you need for each different job you want to run.
Note that there are some differences with the usual runtime environment, see the Help:Toolforge/Build_Service#Known_current_limitations section for details.
Testing locally (optional)
To test if your application will build correctly, you can check on your local computer using pack
. You should be able to build the image and start it, and it should listen on port 8000.
$ pack build --builder tools-harbor.wmcloud.org/toolforge/heroku-builder-classic:22 myimage
$ docker run -p 8000:8000 --rm --entrypoint web myimage
# navigate to http://127.0.0.1:8000 to check that it works
If pack
is not available for your operating system, you can run it via Docker itself.
Note that this is fairly dangerous, as it requires passing the Docker control socket into the container itself,
effectively handing the pack
container full control over your Docker daemon:
$ docker run -u root -v /var/run/docker.sock:/var/run/docker.sock -v "$PWD":/workspace -w /workspace buildpacksio/pack build --builder tools-harbor.wmcloud.org/toolforge/heroku-builder-classic:22 myimage
Configuration
For secrets and other required configuration variables, we recommend using the envvars service, which will allow you to create environment variables with the secret contents and/or configuration.
Example: OAuth with Python + Django
If you followed Help:Toolforge/My first Django OAuth tool, you can migrate that app by extracting the values of the SOCIAL_AUTH_MEDIAWIKI_KEY
and SOCIAL_AUTH_MEDIAWIKI_SECRET
from the environment variables instead.
First let's create the environment variables with the right values:
# Replace xxxxxxxxxxxxx with the real contents
tools.wm-lol@tools-sgebastion-10:~$ toolforge envvars create SOCIAL_AUTH_MEDIAWIKI_SECRET "xxxxxxxxxxxxxxxx"
name value
SOCIAL_AUTH_MEDIAWIKI_SECRET xxxxxxxxxxxxxxx
tools.wm-lol@tools-sgebastion-10:~$ toolforge envvars create SOCIAL_AUTH_MEDIAWIKI_KEY "xxxxxxxxxxxxxxxx"
name value
SOCIAL_AUTH_MEDIAWIKI_KEY xxxxxxxxxxxxxxx
Now you can use it in your settings.py
file like:
import os
SOCIAL_AUTH_MEDIAWIKI_KEY = os.environ.get("SOCIAL_AUTH_MEDIAWIKI_KEY", "dummy-default-value")
SOCIAL_AUTH_MEDIAWIKI_SECRET = os.environ.get("SOCIAL_AUTH_MEDIAWIKI_SECRET", "dummy-default-value")
SOCIAL_AUTH_MEDIAWIKI_URL = 'https://meta.wikimedia.org/w/index.php'
SOCIAL_AUTH_MEDIAWIKI_CALLBACK = 'http://127.0.0.1:8080/oauth/complete/mediawiki/'
Build and deploy
If you are sure that your app will build and start in port 8000, then you can go to login.toolforge.org
, and start a build as your tool. For example:
$ become mytool
$ toolforge build start https://gitlab.wikimedia.org/toolforge-repos/<your-repo>
$ toolforge build show # wait until build passed
See toolforge build start --help
for additional options such as --ref REF
to select a specific branch, tag or commit rather than the current HEAD of the given repository.
Old builds are cleaned up automatically. The system will try to always keep at least one old successful build and a couple of previous failed builds for troubleshooting.
Webservice
To start a web service:
$ toolforge webservice --backend=kubernetes --mount=none buildservice start
Alternatively, put the following in your service.template to make toolforge webservice start
work on its own:
backend: kubernetes
type: buildservice
mount: none
To update the code later, trigger a new build with toolforge build start
as above;
once the build has finished, a normal toolforge webservice restart
will suffice to update it.
To see the logs for your web service, use:
$ toolforge webservice --backend kubernetes buildservice logs -f
Job
To use with the jobs framework:
$ toolforge jobs run --image tool-test/tool-test:latest --command "migrate" --wait --no-filelog some-job
This will run the migrate
command as defined in your Procfile
. You could also pass additional arguments, for example --command "migrate --production"
would run the script specified in Procfile
with the --production
argument.
Supported languages
We currently support all the languages included in Heroku's builder-classic-22 builder:
- Clojure (lein based only for now, until task T338142)
- Go
- Java
- Node.js
- PHP (see Help:Toolforge/My_first_Buildpack_PHP_tool)
- Python (see Help:Toolforge/Build_Service/My_first_Buildpack_Python_tool)
- Ruby
- Scala
Additional documentation for each buildpack, including the list of supported runtime versions, can be found at https://devcenter.heroku.com/articles/buildpacks
Other features
Installing Apt packages
Sometimes you can't get all the libraries you want, or have php and python installed at the same time by using the supported buildpacks only.
In those cases you can install custom packages by creating an Aptfile
at the root of your repository, with the packages you want to install listed one per line (comments are allowed), for example, if you want to install imagemagick
and php
, along with your python application, you can create the following file:
# This file should be placed at the root of your git repository, and named Aptfile
# These will be pulled from the OS repositories
imagemagick
php
mariadb-client=1:10.6.12-0ubuntu0.22.04.1
NOTE: If you use extra packages, you will not be able to build your project locally, so we encourage you to try using your language preferred package manager instead (pip/bundle/npm/composer/...).
Right now the images are based on Ubuntu 22.04 (jammy), although this can change in the future. You can use the packages.ubuntu.com tool to look up available packages and their versions.
Migrating an existing tool
Many tools that are now running in Toolforge Kubernetes should also work with the Build Service with just a few simple changes.
If your tool is hosted on a public Git repository, it's possible the Build Service will automatically detect all that is needed to build a working container image of your tool.
However, it's likely you will have to change a few things:
- You will need to add a Procfile with a `web:` entry to your project specifying the command to start the application (needed even if you are going to be running a job).
- If you're migrating a Python tool that uses uWSGI, replace it with Gunicorn, see the above example.
Currently NFS directories are mounted on the same paths as they were before, the only difference is that for the tool's data directory (/data/project/<toolname>
, instead of using $HOME
, you have to use $TOOL_DATA_DIR
. Right now all tools get NFS mounted, although it's possible that will be configurable with a config option in the near future.
For secrets and most use cases, we recommend using the envvars service, which will allow you to create environment variables with the secret contents and/or configuration.
--mount
argument (which is currently limited to web services only). Currently the default is to mount everything (--mount=all
), to disable mounts you can use --mount=none
. The default might be changed in the future, so it is recommended to explicitely specify --mount=all
if your tool relies on NFS shared storage.Here be dragons: Notes for using NFS with buildpack tools |
---|
The following content has been placed in a collapsed box for improved usability. |
If your tool relied on being run from a certain directory, you'll have to adapt it to run in a different one. (The directory might change depending on which buildpack and builder you're using.) You can use the environment variable Usually this means us one of:
If you have a configuration file under
import os
import yaml
from pathlib import Path
my_config = yaml.safe_load((Path(os.environ["TOOL_DATA_DIR"]) / "myconfig.yaml").read_text())
If you use config_path = 'config.yaml'
if 'TOOL_DATA_DIR' in os.environ:
config_path = os.environ['TOOL_DATA_DIR'] + '/www/python/src/config.yaml'
app.config.from_file(config_path, load=yaml.safe_load, silent=True)
You can skip the |
The above content has been placed in a collapsed box for improved usability. |
Connecting to ToolsDB and WikiReplicas
To connect to the databases, there's two sets of credentials provided through environment variables:
TOOL_DATABASE_USER
andTOOL_DATABASE_PASSWORD
are the user and password to connect to the toolsdb instance.TOOL_REPLICA_USER
andTOOL_REPLICA_PASSWORD
are the user and password to connect to the wikireplicas.
NOTE: Even if the credentials are currently the same for the replicas and toolsdb, we strongly recommend using the variables for each specific service as they might change in the future.
Tutorials for popular languages
We have created some guides (more will be added) on how to deploy apps built with popular languages and frameworks.
Common problems and solutions
Please add to this section any issues you encountered and how you solved them.
Where are my old builds?
In order to be able to maintain a manageable level of builds, we only keep a few ones for each tool.
You should always have the last few successful and failed builds, if that is not the case, please reach to us (see Help:Toolforge/Build_Service#Feedback).
Troubleshooting
We are actively building the debugging capabilities of the system, so they will be improved soon. Some hints:
- If the build failed, try
toolforge build logs
to check the last build logs, you can also try building it locally. - If the build passed, but the webservice does not start or work, try
toolforge webservice --backend kubernetes buildservice logs
to see the logs created by your tool.
If you are unable to figure it out or found a bug, feel free to create a task (see the feedback section) or reach out to us on IRC/etc. (see communications section).
Known current limitations
Multiple language applications (multi-stack) and installing packages
Currently we allow application to install any packages provided by the OS (Ubuntu Jammy at the time of writing), that allows you to have for example PHP installed through apt, and use the Python buildpack for your application.
To install custom packages, follow Help:Toolforge/Build_Service#Installing_Apt_packages.
Note that building assets and an application at the same time is not supported yet (ex. React + flask), you'll have to work around it using scripts or similar (see task T325799 for progress).
Only heroku builder with default languages are supported
We currently support a single builder (heroku-classic) and the default languages and buildpacks shipped with, for the list see: https://devcenter.heroku.com/articles/buildpacks
For this round of the Beta we are focusing on python, but all the other languages supported by the heroku builder are available if you want to help testing and adding support for them (ex. php, ruby, clojure, ...).
No LDAP connection for checking user information
We currently are not using a base image which knows how to use the Developer account LDAP directory. So unix/linux commands that use it, like trying to find the home directory for a given user (expanding ~
) or checking which groups a user is (id <user>
) in will not work as expected.
$HOME not pointing to the tool home
See #Using NFS shared storage above for storage limitations and changes.
Tool home directory is not available
See #Using NFS shared storage above for storage limitations and changes.
Builds aren't automatically triggered upon commit
Ideally a new build would automatically be generated upon pushing a new commit to your tool's git repository. Thus removing the need to manually trigger a build (though the capability to do so would remain). This "trigger" doesn't yet exist.
Out of quota
There's a limited amount of space available to store your builds, there's a recurring job cleaning up old images that runs every 5 minutes (see task T336360), and another one that garbage collects untagged ones every hour, if your build fails and you don't know why, running out of space might be the issue, please open a task or contact us and we'll fix it until the long term fix is rolled out.
Feedback
If you try to use the Build Service, either to deploy a new tool or to migrate an existing one, we would love to hear your feedback!
You can use this Phabricator template to report a bug, or this one for feature requests.
If you find this documentation page lacking in some way, or if you have more general comments, you can leave a comment in the Talk page.
For any other questions you can always reach us through the other communication channels listed at the bottom of this page.
Contributing
To contribute and/or follow up with the development of the project, take a look at the Ongoing Efforts page and the Contributing page.
History
The Build Service was discussed for the first time in 2021. Below are some historical discussions that led to its current design and implementation.
- Phabricator task
- Enhancement proposal: Toolforge push to deploy
- Enhancement proposal: Toolforge Buildpack Implementation
Communication and support
Support and administration of the WMCS resources is provided by the Wikimedia Foundation Cloud Services team and Wikimedia movement volunteers. Please reach out with questions and join the conversation:
- Chat in real time in the IRC channel #wikimedia-cloud connect, the bridged Telegram group, or the bridged Mattermost channel
- Discuss via email after you have subscribed to the cloud@ mailing list
- Subscribe to the cloud-announce@ mailing list (all messages are also mirrored to the cloud@ list)
- Read the News wiki page
Use a subproject of the #Cloud-Services Phabricator project to track confirmed bug reports and feature requests about the Cloud Services infrastructure itself
Read the Cloud Services Blog (for the broader Wikimedia movement, see the Wikimedia Technical Blog)