Using self-hosted Jupyter notebooks

Create a namespace to run the notebook server.

Create a file locally, notebook.yaml, with the following contents. The file contents are described below.

apiVersion: v1
kind: Pod
metadata:
  name: jupyter-notebook
  namespace: namespace
spec:
  containers:
  - name: minimal-notebook
    image: jupyter/all-spark-notebook:lab-3.1.4 # open source image 
    ports:
    - containerPort: 8888
    command: ["start-notebook.sh"]
    args: ["--LabApp.token=''"]
    env:
    - name: JUPYTER_ENABLE_LAB
      value: "yes"
    - name: KERNEL_LAUNCH_TIMEOUT
      value: "400"
    - name: JUPYTER_GATEWAY_URL
      value: "serverUrl"
    - name: JUPYTER_GATEWAY_VALIDATE_CERT
      value: "false"
    - name: JUPYTER_GATEWAY_AUTH_TOKEN
      value: "session-token"

If you are deploying Jupyter notebook to a Fargate-only cluster, label the Jupyter pod with a role label as shown in the following example:

...
metadata:
  name: jupyter-notebook
  namespace: default
  labels:
    role: example-role-name-label
spec:
            ...

Deploy the pod spec to your Amazon EKS cluster:

kubectl apply -f notebook.yaml -n namespace

This will start up a minimal JupyterLab notebook connected to your Amazon EMR on EKS interactive endpoint. Wait until the pod is RUNNING. You can check its status with the following command:

kubectl get pod jupyter-notebook -n namespace

When the pod is ready, the get pod command returns output similar to this:

NAME              READY  STATUS   RESTARTS  AGE
jupyter-notebook  1/1    Running  0         46s

Attach the notebook security group to the node where the notebook is scheduled.

1. First, identify the node where jupyter-notebook pod is scheduled with the describe pod command.

   kubectl describe pod jupyter-notebook -n namespace

2. Open the Amazon EKS console at https://console.amazonaws.cn/eks/home#/clusters.

3. Navigate to the Compute tab for your Amazon EKS cluster and select the node identified by the describe pod command. Select the instance ID for the node.

4. From the Actions menu, select Security > Change security groups to attach the security group that you created in Create a security group.

5. If you are deploying Jupyter notebook pod on Amazon Fargate, create a SecurityGroupPolicy to apply to the Jupyter notebook pod with the role label:

   cat >my-security-group-policy.yaml <<EOF
   apiVersion: vpcresources.k8s.aws/v1beta1
   kind: SecurityGroupPolicy
   metadata:
     name: example-security-group-policy-name
     namespace: default
   spec:
     podSelector:
       matchLabels:
         role: example-role-name-label
     securityGroups:
       groupIds:
         - your-notebook-security-group-id
   EOF

Now, port-forward so that you can locally access the JupyterLab interface:

kubectl port-forward jupyter-notebook 8888:8888 -n namespace

Once that is running, navigate to your local browser and visit localhost:8888 to see the JupyterLab interface:

From JupyterLab, create a new Scala notebook. Here is a sample code snippet that you can run to approximate the value of Pi:

import scala.math.random
import org.apache.spark.sql.SparkSession

/** Computes an approximation to pi */
val session = SparkSession
  .builder
  .appName("Spark Pi")
  .getOrCreate()

val slices = 2
// avoid overflow
val n = math.min(100000L * slices, Int.MaxValue).toInt 
 
val count = session.sparkContext
.parallelize(1 until n, slices)
.map { i =>
  val x = random * 2 - 1
  val y = random * 2 - 1
  if (x*x + y*y <= 1) 1 else 0
}.reduce(_ + _)

println(s"Pi is roughly ${4.0 * count / (n - 1)}")
session.stop()

Use the following command to delete the jupyter-notebook pod:

kubectl delete pod jupyter-notebook -n namespace

Then, delete your interactive endpoint with the delete-managed-endpoint command. For steps to delete an interactive endpoint, see Delete an interactive endpoint. Initially, your endpoint will be in the TERMINATING state. Once all resources have been cleaned up, it transitions to the TERMINATED state.

If you don’t plan to use the notebook security group that you created in Create a security group for other Jupyter notebook deployments, you can delete it. See Delete a security group in the Amazon EC2 User Guide for more information.