Wikimedia Cloud Services team/EnhancementProposals/Decision record T362224 What to use for toolforge components api task execution

Date of the decision: 2024-07-11

People in the decision meeting (alphabetical order):

Decision taken

Option 2 was chosen, with the caveat of moving to option 4 if any blockers are found.

There's open the question for option 2 of how to keep track of the status of the background tasks that we will have to figure out (probably updating the deployment model or similar).

Problem

To be able to manage pipelines in the components api (build 1 + build 2 -> deploy component 1 -> deploy component 2 for example) we need some way to handle execution pipelines.

Constraints and risks

The pipelines don't need to be huge, dozens of components would be the biggest

Options

Option 1

No asynchronous task processing

Pros:

Easiest to implement

Cons:

If the request breaks, the whole pipeline fails

Option 2

After-request asynchronous task processing on the same thread (ex. https://fastapi.tiangolo.com/tutorial/background-tasks/)

Pros:

Easy to implement and setup (no extra components/services/etc. needed)

Cons:

If the service gets restarted (ex. OOM, moving to a different worker) the pipeline breaks

Option 3

Using tekton pipelines

Pros:

Already used for build service
Good pipeline support (made for it)

Cons:

Relatively complex to setup
Pipelines written in tekton yaml + custom image/shell script
Needs interfacing with tekton (similar to what we do with builds service)

Option 4

Using celery with redis

Pros:

Very common pattern
Good pipeline support (made for it)
Pipelines written in the same code than the service

Cons:

Needs a redis instance (we should not reuse the user-facing one for security reasons I think)

Option N

Add you option here!