User:Bstorm/New k8s migration

This page is currently a draft.
Material may not yet be complete, information may presently be omitted, and certain parts of the content may be subject to radical, rapid alteration. More information pertaining to this may be available on the talk page.

As the new, redesigned Kubernetes cluster is deployed to Toolforge, users can begin migrating their web services over to the new system in a self-service fashion. Tools that are migrated should continue functioning exactly as they did in the old cluster with a few important new considerations around the service's quota limits.

Please note that it is entirely possible to migrate back to the old cluster if an issue is discovered until the old cluster is discontinued.

New Quota System

The the new Kubernetes cluster, webservices will have CPU and RAM limits that are lower by default than was done previously. Defaults are set at 50% of a CPU and 512Mi of memory. Users can adjust these up to a certain level (which is the highest level allowed in the old cluster or 1 CPU and 4 GB of RAM) with command line arguments to the webservice command (see information in "how to migrate") or properly formatted Kubernetes specifications (for advanced users). In addition to these limits are quota limits on an entire namespace that determine how many pods can be used, services, total RAM, CPU and others. In the section on how to migrate, commands for interacting with these limits will be provided. The default hard limits for a tool user's entire namespace are:

{
  requests.cpu: 2,
  requests.memory: "6Gi",
  limits.cpu: 2,
  limits.memory: "8Gi",
  pods: 4,
  services: 1,
  services.nodeport: 0, # Nodeport services are not allowed
  replicationcontrollers: 1,
  secrets: 10,
  configmaps: 10,
  persistentvolumeclaims: 3,
}

These limits govern everything a user can create in the new cluster, and you will notice they are a bit higher than seems useful for some things, like pods, but that is to make room for future services. We anticipate founding a process soon to request upgraded limits from the WMCS team for tools that need a bit more to get their work done.

How do I migrate my tool

After **INSERT DATE HERE** the cluster will be ready for you to migrate your tool. After that time, to migrate, follow these steps:

Log into a Toolforge bastion (eg. login.tools.wmflabs.org)
become $yourtool
Run webservice stop and wait a minute or so to make sure it has really stopped.
kubectl config use-context toolforge
Edit your .bash_aliases file (or wherever you keep aliases) to include a line of kubectl=/usr/bin/kubectl so you don't use the older version at /usr/local/bin/kubectl. Then log out and back in so the alias takes.
Launch your web service with the webservice command just like before.
After a moment or so, check to see that things launched successfully at your usual web location (eg. https://tools.wmflabs.org/$mytool).

Example commands to do most of the steps above:

mytool@tools-sgebastion-07:~$ webservice stop && kubectl config use-context toolforge && echo "alias kubectl=/usr/bin/kubectl" >> .bash_aliases && source .bash_aliases
mytool@tools-sgebastion-07:~$ webservice --backend=kubernetes start <type>

If you find that the default resource request limit is not enough for your tool, you can have Kubernetes increase your burstable limit up to a hard limit described in the resources section above. To do that stop your webservice and when you start it up add a --cpu and/or --mem argument (which can be shortened to -c or -m) with an appropriate figure after them. An example is, if my python 3.5 service was dying because it needed more than the default 512Mi of RAM, I could give it 1Gi like:

mytool@tools-sgebastion-07:~$ webservice --backend=kubernetes --mem=1Gi python3.5 start

You can request high CPU, higher memory or both up to the defined limit for your tool.