Amazon EMR on EKS 的 Flink Native Kubernetes 入门 - Amazon EMR
Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅 中国的 Amazon Web Services 服务入门 (PDF)

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

Amazon EMR on EKS 的 Flink Native Kubernetes 入门

Amazon EMR 6.13.0 及更高版本都支持 Flink Native Kubernetes 在 Amazon EKS 集群上运行 Flink 应用程序。要运行 Flink 应用程序,请按照下述步骤操作:

  1. 在使用 Flink Native Kubernetes 命令运行 Flink 应用程序之前,请先完成 设置 Amazon EMR on EKS 的 Flink Native Kubernetes 中的步骤。

  2. 下载并安装 Flink

  3. 设置以下环境变量的值。

    #Export the FLINK_HOME environment variable to your local installation of Flink export FLINK_HOME=/usr/local/bin/flink #Will vary depending on your installation export NAMESPACE=flink export CLUSTER_ID=flink-application-cluster export IMAGE=<123456789012.dkr.ecr.sample-Amazon Web Services 区域-.amazonaws.com/flink/emr-6.13.0-flink:latest> export FLINK_SERVICE_ACCOUNT=emr-containers-sa-flink export FLINK_CLUSTER_ROLE_BINDING=emr-containers-crb-flink
  4. 创建服务账户来管理 Kubernetes 资源。

    kubectl create serviceaccount $FLINK_SERVICE_ACCOUNT -n $NAMESPACE kubectl create clusterrolebinding $FLINK_CLUSTER_ROLE_BINDING --clusterrole=edit --serviceaccount=$NAMESPACE:$FLINK_SERVICE_ACCOUNT
  5. 运行 run-application CLI 命令。

    $FLINK_HOME/bin/flink run-application \ --target kubernetes-application \ -Dkubernetes.namespace=$NAMESPACE \ -Dkubernetes.cluster-id=$CLUSTER_ID \ -Dkubernetes.container.image.ref=$IMAGE \ -Dkubernetes.service-account=$FLINK_SERVICE_ACCOUNT \ local:///opt/flink/examples/streaming/Iteration.jar 2022-12-29 21:13:06,947 INFO org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment requires a fixed port. Configuration blob.server.port will be set to 6124 2022-12-29 21:13:06,948 INFO org.apache.flink.kubernetes.utils.KubernetesUtils [] - Kubernetes deployment requires a fixed port. Configuration taskmanager.rpc.port will be set to 6122 2022-12-29 21:13:07,861 WARN org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Please note that Flink client operations(e.g. cancel, list, stop, savepoint, etc.) won't work from outside the Kubernetes cluster since 'kubernetes.rest-service.exposed.type' has been set to ClusterIP. 2022-12-29 21:13:07,868 INFO org.apache.flink.kubernetes.KubernetesClusterDescriptor [] - Create flink application cluster flink-application-cluster successfully, JobManager Web Interface: http://flink-application-cluster-rest.flink:8081
  6. 检查创建好的 Kubernetes 资源。

    kubectl get all -n <namespace> NAME READY STATUS RESTARTS AGE pod/flink-application-cluster-546687cb47-w2p2z 1/1 Running 0 3m37s pod/flink-application-cluster-taskmanager-1-1 1/1 Running 0 3m24s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/flink-application-cluster ClusterIP None <none> 6123/TCP,6124/TCP 3m38s service/flink-application-cluster-rest ClusterIP 10.100.132.158 <none> 8081/TCP 3m38s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/flink-application-cluster 1/1 1 1 3m38s NAME DESIRED CURRENT READY AGE replicaset.apps/flink-application-cluster-546687cb47 1 1 1 3m38s
  7. 端口转发到 8081。

    kubectl port-forward service/flink-application-cluster-rest 8081 -n <namespace> Forwarding from 127.0.0.1:8081 -> 8081
  8. 在本地访问 Flink UI。

    访问 Flink UI。
  9. 删除 Flink 应用程序。

    kubectl delete deployment.apps/flink-application-cluster -n <namespace> deployment.apps "flink-application-cluster" deleted

有关向 Flink 提交应用程序的更多信息,请参阅 Apache Flink 文档中的 Native Kubernetes