Portal:Toolforge/Admin/System Overview

This page contains a system overview of Toolforge from the engineering point of view. Find here high level information of the different bits and pieces that make Toolforge what it is.

Summary

Toolforge is a Platform as a Service. It is meant to offer pre-made and managed computing facilities to its users.

At the core of its service, there is a computing engine. Well, as of this writing there are two, Grid Engine (to be deprecated) and Kubernetes. Users log in into Toolforge via SSH and deploy tools (mainly webservices and jobs) into one of the computing engines.

The About Toolforge user-oriented page contains more information on the basic aspects of the service.

All pieces of Toolforge are deployed inside a Cloud VPS project (or tenant) called tools. The staging/development project is called toolsbeta.

Use cases

This section captures the most common use cases that Toolforge supports.

webservices

TODO.

jobs

TODO.

They key components

Information on the several key pieces that make Toolforge what it is.

LDAP

This is where user accounts are stored.

Bastions

Bastion servers are used by toolforge user as the entry point for the service. They are Cloud VPS virtual machines that allow SSH connections from the internet. Usual FQDNs are login.toolforge.org and dev.toolforge.org.

A strong shared-system policy is enforced to ensure that users don't easily hog system resources (like CPU or RAM).

NFS

As of today, NFS is at the core of Toolforge. We use it for many purposes, including:

• to store toolforge user's home directories (remember, we allow SSH to the bastions)
• to store toolforge user's source code and distribute it to the computing backends (grid engine or kubernetes)
• to store toolforge user's credentials, like kubernetes TLS certs or wiki replicas database account credentials
• to store toolforge user's logs, generated at runtime from webservices and jobs
• to distribute wiki dumps
• to store toolforge tool temp files and similar

Please note that NFS is an old technology, and we have been working for years in reducing dependency on it with the ultimate goal of stop using NFS at all in the future.

Kubernetes

We have a particular Kubernetes deployment, which is described in its own page.

There are a number of custom components deployed into Toolforge kubernetes, see Portal:Toolforge/Admin/Kubernetes/Custom_components.

RBAC and security

See main page: Portal:Toolforge/Admin/Kubernetes/RBAC_and_PSP.

jobs framework

See main page: Portal:Toolforge/Admin/Kubernetes/Jobs_framework.

Network

Toolforge has a particular network setup, see Portal:Toolforge/Admin/Kubernetes/Networking_and_ingress.

See also Portal:Toolforge/Admin/Dynamicproxy.

Other components

There are a number of components that are part of the Toolforge offering beyond the key building blocks.

striker

The entry point for account creation.

TODO.

databases

TODO. Talk here about.

• toolsdb
• wiki replicas

Some history

TODO:

• toolserver
• grid-only system

See also

TODO.