Kubernetes/Kubernetes Workshop/Load Testing

From Wikitech

Overview

At the end of this module, you should be able to:

  • Carry out load testing using k6.io.

Remote Testing with k6.io

This module uses a program like apachebench (in package apache2-utils) or hey to generate load on a service. These programs then report back requests, errors, latencies, and latency histograms per second. You will use the calculator-service in Module 10 for this walk-through.

Minikube usually loads Docker images from Docker Hub or any other repository, but you can use a local image if you set the ImagePullPolicy to Never in the values.yaml file.

You can build a local Docker image for the calculator-service for easier manipulation and faster updates. Take the following steps:

  • Clone the calculator-service repository from Gerrit.
$ minikube start
$ eval $(minikube docker-env)
$ docker build --tag wmfcalc-mk3 .
$ docker run -p 8080:8080 wmfcalc-mk3

The calculator service is simple and should not present significant CPU usage. Start with a low CPU allocation in Kubernetes: CPU = 100m. calculator-service reports its memory footprint on its /metrics page, which is relatively low (<14 MB). You can set memory to 16 Mb. Set both variables in the deployment file.

deployment.yaml 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: calc
  labels:
    app: calc
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
  selector:
    matchLabels:
      app: calc
  template:
    metadata:
      labels:
        app: calc
    spec:
      containers:
       - name: calc
         image: docker-registry.wikimedia.org/wikimedia/blubber-doc-example-calculator-service:stable
         imagePullPolicy: Always
         resources:
           requests:
             memory: "16Mi"
             cpu: "100m"
           limits:
             memory: "16Mi"
             cpu: "100m"

Create a deployment and make curl requests: 

$ kubectl create -f deployment.yaml
$ curl localhost:8080/api?2+3 # to test
{"operation":"2+3","result":"5"}
$ curl http://<ip:port>/metrics # to get metrics
$ curl http://<ip:port>/api?2+38+(44)(64)((66555/2-3)) # more complex
$ curl http://<ip:port>/api?2+38+(44)(64)((66^555/2-3)) # syntax error
$ ab -q -n 1000 -c 8 http://<ip:port>/api?2+3 | grep Requests #Requests per second: 168.51 [#/sec] (mean)
$ curl http://<ip:port>/metrics
$ curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa #ength error input is limited to 60 characters

Note:

  • Start with one replica for a baseline, take note of the requests per second, then increase the number of replicas to see whether the service scales.
  • Start the service and deployment.
  • Then run tests with ab - simple math expression, a more complicated one with a parsing error, and one that induces the length error.

Assessing Baseline Metrics

About 80 rps - memory is at 13 MB - see the Appendix for more data. When you increase the number of replicas to 2, 4, 8, then 16, it results in a plateau at 1000 rps with ab, but the hey binary seems to get more requests up to the 1900 rps area.

Overall a configuration with two replicas using 100m CPU and 16 Mb memory will serve 150+ rps and have some high availability and is a good baseline for our first release.

Local Testing with k6.io

k6.io is an external testing service. You can record test sequences with a browser or code them using Javascript. k6.io has test execution machines worldwide.

The service to test needs to be externally available. In this test, you will use your Kubernetes (k8s) installation from Wikimedia Cloud Service (WMCS) from Module 6, but in addition, you need a floating IP to provide external access.

  1. Take the following steps:
  2. Install k8s on WMCS (see chapter 6)
  3. Deploy calc/1.2
  4. Define a service using a nodeport. Find the port mapped (30000+)
  5. Map the floating IP to one of the nodes1
  6. Open the firewall for port from Step 3
  7. Test access from the Internet:
$ curl http://{floating ip}:{port}/api?2+6 and http://{floating ip}:{port}/metrics
  1. Log in to k6.io. (A trial account would suffice)
  2. Run a script to test calc. Here is an example created from k6.io's library. The test is as follows: a 5-minute test, 1 minute ramping up to 20 Virtual users, then 3 minutes at 20 Virtual Users, then ramping down. The test is a simple get, followed by a 1-second sleep call.
import { sleep } from 'k6'
import http from 'k6/http'

// See https://k6.io/docs/using-k6/options
export const options = {
  stages: [
    { duration: '1m', target: 20 },
    { duration: '3m', target: 20 },
    { duration: '1m', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.02'], // http errors should be less than 2%
    http_req_duration: ['p(95)<2000'], // 95% requests should be below 2s
  },
  ext: {
    loadimpact: {
      distribution: {
        'amazon:us:ashburn': { loadZone: 'amazon:us:ashburn', percent: 100 },
      },
    },
  },
}

export default function main() {
  let response = http.get('http://185.15.56.95:30162/api?2+3')
  sleep(1)
}

Output Analysis

  • Output from calc's metrics collection:
$ curl http://185.15.56.95:30162/metrics
start_time 1618522447.9971263
wellformed_total{method="get"} 23669
wellformed_total{method="post"} 0
nonwellformed_total 0
memory 18292736
duration_bucket{le="1"} 6
duration_bucket{le="2"} 562
duration_bucket{le="4"} 11376
duration_bucket{le="8"} 11488
duration_bucket{le="16"} 200
duration_bucket{le="32"} 27
duration_bucket{le="32+"} 13
  • More test runs:

50 users from São Paulo, Brazil

50 users from Mumbai, India

Appendix:

One Replica

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    129.23 [#/sec] (mean)

ab -q -n 1000 -c 2 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    159.30 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    42.55 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    164.13 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    138.44 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+3 | grep Requests
Requests per second:    168.51 [#/sec] (mean)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))
{"operation":"2+38+(44)(64)((66555/2-3))","result":"7031834.0"}

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    93.77 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    106.45 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    104.89 [#/sec] (mean)

## Parse error (^ is not supported)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3))
{"operation":"2+38+(44)(64)((66555/2-3))","result":"None"}

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)*((66555/2-3)) | grep Requests
Requests per second:    86.60 [#/sec] (mean)

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3)) | grep Requests
Requests per second:    95.75 [#/sec] (mean)

ab -q -n 1000 -c 8 http://192.168.49.2:32459/api?2+38+(44)(64)((66^555/2-3)) | grep Requests
Requests per second:    95.08 [#/sec] (mean)

## Length error (input is limited to 60 characters)

curl http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa

ab -q -n 1000 -c 1 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3))aaaaaaaaaaaa| grep Requests
Requests per second:    195.40 [#/sec] (mean)

curl http://192.168.49.2:32459/metrics
start_time 1613141183.1434238
wellformed_total{method="get"} 1000
wellformed_total{method="post"} 0
nonwellformed_total 2000
memory 13492224
duration_bucket{le="1"} 0
duration_bucket{le="2"} 0
duration_bucket{le="4"} 994
duration_bucket{le="8"} 209
duration_bucket{le="16"} 628
duration_bucket{le="32"} 40
duration_bucket{le="32+"} 129

## 4 replicas

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    410.19 [#/sec] (mean)

## 8 replicas

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    954.36 [#/sec] (mean)

## 16 replicas - 1st run too fast 2000+ rps seemed suspicious - increase number of  requests

ab -q -n 1000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    2173.27 [#/sec] (mean)

ab -q -n 10000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1132.95 [#/sec] (mean)

ab -q -n 20000 -c 4 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1011.18 [#/sec] (mean)

## 24 replicas

ab -q -n 20000 -c 16 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
Requests per second:    1272.75 [#/sec] (mean)

hey -n 20000 -c 16 http://192.168.49.2:32459/api?2+38+(44)(64)((66555/2-3)) | grep Requests
 Requests/sec: 1941.3902
Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Kubernetes/Kubernetes_Workshop/Load_Testing&oldid=1990124"