OpenSearch on K8s test deploy

Applying

1. Clone this gerrit change to your homedir. That will get you the latest version of the chart. Note: If you click the ⠇in the upper-right corner of the Gerrit web UI, then click "Download patch", it will give you the exact commands you need to reset, cherry-pick, etc.

2. Cherry-pick this change on top of it. This will get you the helmfile and release-specific values.

3. Edit the following files as needed

helmfile.d/dse-k8s-services/_opensearch_common_/values-cluster.yaml Contains values that apply to all clusters. You can make one-off changes here.
helmfile.d/dse-k8s-services/opensearch-test/helmfile.yaml Point helmfile the homedir version of your chart instead of the production version, for example:

-    chart: wmf-stable/opensearch-cluster
+    chart: ../../../charts/opensearch-cluster

4. As root, navigate to your homedir deploy and apply:

root@deploy2002:/home/bking/deployment-charts/helmfile.d/dse-k8s-services/opensearch-test# helmfile -e dse-k8s-eqiad -i apply --selector name=cluster

What Should Happen When You Apply

0. The only pods in the namespace are the operator:

kubectl get po
NAME                                                              READY   STATUS      RESTARTS   AGE
operator-opensearch-operator-controller-manager-54cc5d7965flgrc   1/1     Running     0          7d20h

1. The bootstrap, first master, and security pods are created:

NAME                                                              READY   STATUS    RESTARTS   AGE
cluster-bootstrap-0                                               1/1     Running   0          61s
cluster-masters-0                                                 0/1     Running   0          61s
cluster-securityconfig-update-f7wmp                               1/1     Running   0          61s
operator-opensearch-operator-controller-manager-54cc5d7965flgrc   1/1     Running   0          7d19h

2. The cluster is fully deployed:

kubectl get po
NAME                                                              READY   STATUS      RESTARTS   AGE
cluster-masters-0                                                 1/1     Running     0          63m
cluster-masters-1                                                 1/1     Running     0          62m
cluster-masters-2                                                 1/1     Running     0          60m
cluster-securityconfig-update-f7wmp                               0/1     Completed   0          63m
operator-opensearch-operator-controller-manager-54cc5d7965flgrc   1/1     Running     0          7d20h

Testing

Pane 1

kubectl port-forward --address 0.0.0.0 cluster-masters-0 :9200
Forwarding from 0.0.0.0:45679 -> 9200

Pane 2

set variables

You can find the username and password in /srv/git/private/hieradata/role/common/deployment_server/kubernetes.yaml on the Puppet server.

PW="user:password" <- used for basic auth with cURL. 

PT=45679 <- connect to the same port number from pane 1

with authentication

curl -ks -u ${PW} https://0:${PT}/_prometheus/metrics

without authentication

curl -ks https://0:${PT}/_cat/nodes
curl -ks  https://0:${PT}/_cat/health
curl -ks https://0:${PT}/_prometheus/metrics

Destroying

1. As root, navigate to your homedir deploy and destroy:

root@deploy2002:/home/bking/deployment-charts/helmfile.d/dse-k8s-services/opensearch-test# helmfile -e dse-k8s-eqiad -i destroy --selector name=cluster

2. Wait a minute or so, then delete the volume claims:

for n in {0..2}; do kubectl delete pvc data-cluster-masters-${n}; done

If you don't do this, the next time you deploy a cluster, the old PVs will be used. The bootstrap cluster will detect the previous cluster's data and refuse to create the cluster.