Data Platform/Systems/Spark/Kubernetes

We are currently working on enabling Spark jobs on the dse-k8s Kubernetes clusters, by means of the spark-operator.

This means that Kubernetes has support for two new custom resources named SparkApplication and ScheduledSparkApplication.

When these resources are created, the operator executes `spark-submit` with the appropriate parameters, using Spark's built-in Kubernetes support.

A driver pod is created in the target namespace and the driver then launches the specificed number of executor pods to carry out the required work.

The user guide for working with Spak applications on Kubernetes is here: https://kubeflow.github.io/spark-operator/docs/user-guide.html