Proposal For Complex, Multi-Service Development Ecosystem

This proposal was for a kubernetes based complex dev environment, but we are going ahead with docker-compose instead.

Background

Developing MediaWiki and services is difficult for developers because they don’t have a standard, well maintained environment that is easy to use and understand, and fulfills the needs of multiple stakeholders (primary: the various kinds of developers, secondary: QA, product) while also providing production-like conditions.

Summary of Proposed Solution

The complex, multi-service development ecosystem will be provided in the form of a virtual machine image that can be used by any hypervisor to create a virtual machine. We will use a virtual machine in order to be platform independent. The virtual machine will be used to run containers of the services desired by the user.

Helm charts will be utilized to deploy containers to kubernetes, whether in the virtual machine development environment, or a cloud-based test environment. A proof-of-concept kubernetes environment was created to demonstrate the feasibility of a kubernetes-based environment, which exposed some focus areas addressed by the requirements in this proposal.

Why Use Containers?

Containers will be used in order to ensure an isolated environment that can match what is running in production as closely as possible. User changes to the environment can be easily wiped away with a fresh container. We can also provide the relevant base images that will allow developers to test different versions of servers, etc. Users can also reproduce a set of changes from another user more easily. RelEng/SRE is working to make containers the standard artifact for promotion of code through the deployment pipeline. As a result, we want to unify production and development by making sure developers can run and test their work in containers. This approach will allow us to unify the test and development environments, reducing the mental load on developers to understand multiple different systems.

Why Use Kubernetes?

Kubernetes will allow us to more easily orchestrate changes to the container ecosystem as a whole, and make it easier to network between services. It also aims to have more parity with the future state of production.

Minimal Role-based System

Unlike mediawiki-vagrant, there will be no role-based dependency management. Dependency management is not a problem to be solved by environment configuration.

The roles increase the complexity of the environment and the coupling between services and the development environment makes maintenance very difficult. It’s our philosophy that developers should not have to do extra work to maintain a development environment when publishing changes to their services. If there are roles, they may be a small set of configurations enabling a default set of services/extensions for a type of developer.

Software

The virtual machine image will come with the following pre-installed:

Kubernetes cluster - For running the containerized services
kubectl - to interact with k8s cluster
helm - to install helm charts
development environment git repository???

Updating The Software

When it is desired to update to a newer version of software, Release Engineering will provide a new virtual machine image. It would also be acceptable for developers to run the updates on their virtual machine if desired.

Developing

Sharing Files

Users may choose to share files from the host machine so that they can develop using their preferred tools without having to go through installing them on the virtual machine. There must be an easy way for users to share the files from the host to the virtual machine, for example, sshfs.

Service Access

Users should be able to easily access the services running on the virtual machine, Provide resolution for each service exposed in k8s (make it easy to access, like localhost)

Docker Images

The public Wikimedia registry should provide two categories of docker images for development environments:

Base docker images - The base docker images will include everything needed to run code and tests under development so that a user can mount their git repository into the container.
Docker images for each build - Each time a commit is submitted, the build process should create a docker image based on that change and upload it to the public registry. These images will be used by developers for creating development environments when they need to test a combination of changes to services.

Collaborating

In this model, it’s easy for developers to share their environment configuration. They just need to share their helm values file with another developer in order for them to reproduce the environment.

Viewing Changes

Although developers can view changes by downloading each patch and running it, it might be harder for others who don’t have enough background knowledge to see changes before they are deployed to production.

Test Environment

Provide a way to deploy a set of changes to a fresh test environment. This could be in the form of a patchset to a helm charts repo updating the image tags for the desired services. When the patchset is created or updated, an event could trigger the helm chart to be installed to a namespace in a kubernetes cluster. In addition, a GUI could be created to allow users to insert image tags for their services and create a patchset for them. When the patchset is merged (indicating the original changes to services were accepted and merged), another event could trigger deletion of the namespace

Example Test Environment Workflow:

This workflow assumes a change to one project. If there are changes to multiple projects meant to be tested together, something like the depends-on message in the git commit could be used to determine whether or not to create a new patchset to the environment helm charts or update an existing one.

Documentation

Documentation on usage development on the development environment should be up to date and easily discoverable. Ideally, some form of documentation will exist in version control so that it can be tied to the source. Tutorials and other guidance would be helpful, especially since most developers might not know how to use Kubernetes or Helm. Assistance should be provided to those who need help running or making changes to the environment.

Open Questions

Should the git repositories be included in the vm image, or should they be shared from the host?
Want to reduce the amount of extra knowledge people need to have to run the dev env. If there is a CLI for interacting with the environment, what should it enable people to do?

Challenges

Publishing an image for each commit might take up too many resources
How to use IDEs in conjunction with the environment for things like running tests?
Need to determine whether also needing to share config files for some services will be problematic. How will this work in the test environment?

Arguments Against

Kubernetes vs docker-compose: Is it sensible to design a solution around a production environment that doesn’t yet fully exist?
Is this solution easily usable/hackable? Most devs might not have experience with tools like Helm and Kubernetes, which could make adoption and customization more difficult.