From Wikitech
Jump to navigation Jump to search

This page contains information for common maintenance tasks we administrator do in Toolforge.

Is the Toolforge-specific version of the CloudVPS admin maintenance conterpart wikipage.

Admin scripts

This section contains documentation on several Toolforge-specific admin scripts we have.


This script helps manage Grid Engine exec nodes by allowing for depool, repool, and check status, given an exec node hostname.

Usage is:

  • exec-manage [status|depool|repool] exec_host_name
  • exec-manage [list]


root@tools-sge-master-01:~# exec-manage status tools-sgeexec-1001.tools.eqiad.wmflabs


This script is for the new Toolforge kubernetes cluster.

It creates a x509 cert for a given ServiceAccount that can be used to auth against the k8s API. It must be executed from a control plane node.

user@tools-k8s-control-3:~$ sudo wmcs-k8s-get-cert -help
INFO: Usage of this script:
      /usr/local/sbin/wmcs-k8s-get-cert -h/--help   	- show help and exit
      /usr/local/sbin/wmcs-k8s-get-cert <svcname>   	- generate a x509 TLS cert from the kubernetes API
      /usr/local/sbin/wmcs-k8s-get-cert <svcname> -v	- same, but in verbose mode

Example usage:

user@tools-k8s-control-3:~ $ sudo -i wmcs-k8s-get-cert myserviceaccount

NOTE: x509 certs expire in 1 year usually. See also Portal:Toolforge/Admin/Kubernetes/Certificates


This is a script for the Toolforge Kubernetes cluster, deployed to the control plane nodes.

It will give a specified tool account a service account named $tool-obs that will allow a pod running as it to have "view" access to the entire cluster. This is for building tools similar to Openstack Browser for Kubernetes.

user@tools-k8s-control-3:~$ sudo -i  # You'll want full root for k8s credentials
root@tools-k8s-control-3:~# wmcs-enable-cluster-monitor <tool-name>

If that runs successfully, you'll want to run the tool as that service account using a script like k8s_webservice.sh instead of using webservice.

Important point about using a service account: it must use the credentials mounted at /var/secrets/kubernetes.io/serviceaccount in the pod and not $HOME/.kube/config or you'll get the tool permissions instead of the $tool-obs permissions. This is largely because of invoking the pod presets to get the $HOME variable set to /data/project/$tool and the NFS mounted.


This script is for the Toolforge kubernetes cluster.

It creates a k8s secret to hold x509 cert for a given ServiceAccount which you can use inside a pod to auth to the k8s API server.

user@tools-k8s-control-3:~$ sudo wmcs-k8s-secret-for-cert -h
INFO: this script creates a k8s secret to hold a x509 cert for a
      given service account (pub/priv) which you can then use inside
      a pod, mounting the secret as a volume. Usage:

  /usr/local/sbin/wmcs-k8s-secret-for-cert [-n <namespace>] -s <secretname> -a <svcname> [-v] | -h

    -n namespace    the namespace in which the secret will be created (uses default if not specified).
    -s secretname   name of the secret being created. This is the name you use to mount it later on.
    -a svcname      service name, doing RBAC auth for this service account.
    -v              verbose mode.

    -h              show help and exit.

Example usage:

user@tools-k8s-control-3:~ $ sudo -i wmcs-k8s-secret-for-cert -n kube-system -s mysecret -a myserviceaccount
secret/mysecret created

This script is calling internally other admin script, wmcs-k8s-get-cert.

When the secret is created, it can be used in a pod like this:

apiVersion: apps/v1
kind: Deployment
  name: my-deployment
  namespace: my-namespace
      name: my-whatever
      serviceAccountName: my-serviceaccount
      - name: my-secret-cert
          secretName: my-secret-cert
      - name: my-pod
        image: whatever
          - --tls-cert-file=/etc/certs/cert.pem
          - --tls-private-key-file=/etc/certs/key.pem
        - name: my-secret-cert
          mountPath: /etc/certs
          readOnly: true

NOTE: X509 certs expire in 1 year. See also Portal:Toolforge/Admin/Kubernetes/Certificates


This script is used to chain all the required steps required to upgrade the farm of k8s worker nodes on a given kubeadm-based kubernetes cluster.

More info in Portal:Toolforge/Admin/Kubernetes/Upgrading_Kubernetes.

Other scripts

Other scripts you may find interesting.


This script is used to build and publish packages into aptly. More information in the packaging page.

You can locate it in the puppet tree: modules/toolforge/files/wmcs-package-build.py. It is intended to run from your laptop.