Getting started with the Spark operator for Amazon EMR on EKS

If you haven't already, complete the steps in Setting up the Spark operator for Amazon EMR on EKS.

Authenticate your Helm client to the Amazon ECR registry. In the following command, replace the region-id values with your preferred Amazon Web Services Region, and the corresponding ECR-registry-account value for the Region from the Amazon ECR registry accounts by Region page.

aws ecr get-login-password \
--region region-id | helm registry login \
--username AWS \
--password-stdin ECR-registry-account.dkr.ecr.region-id.amazonaws.com

Install the Spark operator with the following command.

For the Helm chart --version parameter, use your Amazon EMR release label with the emr- prefix and date suffix removed. For example, with the emr-6.12.0-java17-latest release, specify 6.12.0-java17. The example in the following command uses the emr-7.2.0-latest release, so it specifies 7.2.0 for the Helm chart --version.

helm install spark-operator-demo \
  oci://895885662937.dkr.ecr.region-id.amazonaws.com/spark-operator \
  --set emrContainers.awsRegion=region-id \
  --version 7.2.0 \
  --namespace spark-operator \
  --create-namespace 

By default, the command creates service account emr-containers-sa-spark-operator for the Spark operator. To use a different service account, provide the argument serviceAccounts.sparkoperator.name. For example:

--set serviceAccounts.sparkoperator.name my-service-account-for-spark-operator

If you want to use vertical autoscaling with the Spark operator, add the following line to the installation command to allow webhooks for the operator:

--set webhook.enable=true

Verify that you installed the Helm chart with the helm list command:

helm list --namespace spark-operator -o yaml

The helm list command should return your newly-deployed Helm chart release information:

app_version: v1beta2-1.3.8-3.1.1
chart: spark-operator-7.2.0
name: spark-operator-demo
namespace: spark-operator
revision: "1"
status: deployed
updated: 2023-03-14 18:20:02.721638196 +0000 UTC

Complete installation with any additional options that you require. For more informtation, see the spark-on-k8s-operator documentation on GitHub.

Before you can run a Spark application with the Spark operator, complete the steps in Setting up the Spark operator for Amazon EMR on EKS and Install the Spark operator.

Create a SparkApplication definition file spark-pi.yaml with the following example contents:

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-pi
  namespace: spark-operator
spec:
  type: Scala
  mode: cluster
  image: "895885662937.dkr.ecr.us-west-2.amazonaws.com/spark/emr-6.10.0:latest"
  imagePullPolicy: Always
  mainClass: org.apache.spark.examples.SparkPi
  mainApplicationFile: "local:///usr/lib/spark/examples/jars/spark-examples.jar"
  sparkVersion: "3.3.1"
  restartPolicy:
    type: Never
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/tmp"
        type: Directory
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    labels:
      version: 3.3.1
    serviceAccount: emr-containers-sa-spark
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    labels:
      version: 3.3.1
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"

Now, submit the Spark application with the following command. This will also create a SparkApplication object named spark-pi:

kubectl apply -f spark-pi.yaml

Check events for the SparkApplication object with the following command:

kubectl describe sparkapplication spark-pi --namespace spark-operator