对使用 PersistentVolumeClaims(PVC)的任务进行问题排查 - Amazon EMR
Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅 中国的 Amazon Web Services 服务入门 (PDF)

对使用 PersistentVolumeClaims(PVC)的任务进行问题排查

如果您需要创建、列出或删除作业的 PersistentVolumeClaims (PVC),但不向默认 Kubernetes 角色 emr-containers 添加 PVC 权限,则在您提交作业时,作业会失败。如果没有这些权限,则 emr-containers 角色无法为 Spark 驱动程序或 Spark 客户端创建必要的角色。如错误消息所示,向 Spark 驱动程序或客户端角色添加权限是不够的。emr-containers 主角色还必须包括所需的权限。本节介绍如何向 emr-containers 主角色添加所需的权限。

验证

要验证您的 emr-containers 角色是否具有必要的权限,请使用您自己的值设置 NAMESPACE 变量,然后运行以下命令:

export NAMESPACE=YOUR_VALUE kubectl describe role emr-containers -n ${NAMESPACE}

此外,若要验证 Spark 和客户端角色是否具有必要的权限,请运行以下命令:

kubectl describe role emr-containers-role-spark-driver -n ${NAMESPACE} kubectl describe role emr-containers-role-spark-client -n ${NAMESPACE}

如果没有权限,请按如下方式继续修补。

修补

  1. 如果没有权限的作业当前正在运行,则停止这些作业。

  2. 创建名为 RBAC_Patch.py 的文件,如下所示:

    import os import subprocess as sp import tempfile as temp import json import argparse import uuid def delete_if_exists(dictionary: dict, key: str): if dictionary.get(key, None) is not None: del dictionary[key] def doTerminalCmd(cmd): with temp.TemporaryFile() as f: process = sp.Popen(cmd, stdout=f, stderr=f) process.wait() f.seek(0) msg = f.read().decode() return msg def patchRole(roleName, namespace, extraRules, skipConfirmation=False): cmd = f"kubectl get role {roleName} -n {namespace} --output json".split(" ") msg = doTerminalCmd(cmd) if "(NotFound)" in msg and "Error" in msg: print(msg) return False role = json.loads(msg) rules = role["rules"] rulesToAssign = extraRules[::] passedRules = [] for rule in rules: apiGroups = set(rule["apiGroups"]) resources = set(rule["resources"]) verbs = set(rule["verbs"]) for extraRule in extraRules: passes = 0 apiGroupsExtra = set(extraRule["apiGroups"]) resourcesExtra = set(extraRule["resources"]) verbsExtra = set(extraRule["verbs"]) passes += len(apiGroupsExtra.intersection(apiGroups)) >= len(apiGroupsExtra) passes += len(resourcesExtra.intersection(resources)) >= len(resourcesExtra) passes += len(verbsExtra.intersection(verbs)) >= len(verbsExtra) if passes >= 3: if extraRule not in passedRules: passedRules.append(extraRule) if extraRule in rulesToAssign: rulesToAssign.remove(extraRule) break prompt_text = "Apply Changes?" if len(rulesToAssign) == 0: print(f"The role {roleName} seems to already have the necessary permissions!") prompt_text = "Proceed anyways?" for ruleToAssign in rulesToAssign: role["rules"].append(ruleToAssign) delete_if_exists(role, "creationTimestamp") delete_if_exists(role, "resourceVersion") delete_if_exists(role, "uid") new_role = json.dumps(role, indent=3) uid = uuid.uuid4() filename = f"Role-{roleName}-New_Permissions-{uid}-TemporaryFile.json" try: with open(filename, "w+") as f: f.write(new_role) f.flush() prompt = "y" if not skipConfirmation: prompt = input( doTerminalCmd(f"kubectl diff -f {filename}".split(" ")) + f"\n{prompt_text} y/n: " ).lower().strip() while prompt != "y" and prompt != "n": prompt = input("Please make a valid selection. y/n: ").lower().strip() if prompt == "y": print(doTerminalCmd(f"kubectl apply -f {filename}".split(" "))) except Exception as e: print(e) os.remove(f"./{filename}") if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument("-n", "--namespace", help="Namespace of the Role. By default its the VirtualCluster's namespace", required=True, dest="namespace" ) parser.add_argument("-p", "--no-prompt", help="Applies the patches without asking first", dest="no_prompt", default=False, action="store_true" ) args = parser.parse_args() emrRoleRules = [ { "apiGroups": [""], "resources": ["persistentvolumeclaims"], "verbs": ["list", "create", "delete"] } ] driverRoleRules = [ { "apiGroups": [""], "resources": ["persistentvolumeclaims"], "verbs": ["list", "create", "delete"] }, { "apiGroups": [""], "resources": ["services"], "verbs": ["get", "list", "describe", "create", "delete", "watch"] } ] clientRoleRules = [ { "apiGroups": [""], "resources": ["persistentvolumeclaims"], "verbs": ["list", "create", "delete"] } ] patchRole("emr-containers", args.namespace, emrRoleRules, args.no_prompt) patchRole("emr-containers-role-spark-driver", args.namespace, driverRoleRules, args.no_prompt) patchRole("emr-containers-role-spark-client", args.namespace, clientRoleRules, args.no_prompt)
  3. 运行 Python 脚本:

    python3 RBAC_Patch.py -n ${NAMESPACE}
  4. 显示新权限和旧权限之间的 kubectl 差异。按 y 以修补角色。

  5. 使用附加权限验证三个角色,如下所示:

    kubectl describe role -n ${NAMESPACE}
  6. 运行 Python 脚本:

    python3 RBAC_Patch.py -n ${NAMESPACE}
  7. 运行命令后,将显示新旧权限之间的 kubectl 差异。按 y 以修补角色。

  8. 使用附加权限验证三个角色:

    kubectl describe role -n ${NAMESPACE}
  9. 再次提交作业。

手动修补

如果您的应用程序所需的权限适用于 PVC 规则以外的其他内容,您可以根据需要为 Amazon EMR 虚拟集群手动添加 Kubernetes 权限。

注意

角色 emr-containers 是主要角色。这意味着它必须提供所有必要的权限,然后您才能更改底层驱动程序或客户端角色。

  1. 通过运行以下命令将当前权限下载到 yaml 文件:

    kubectl get role -n ${NAMESPACE} emr-containers -o yaml >> emr-containers-role-patch.yaml kubectl get role -n ${NAMESPACE} emr-containers-role-spark-driver -o yaml >> driver-role-patch.yaml kubectl get role -n ${NAMESPACE} emr-containers-role-spark-client -o yaml >> client-role-patch.yaml
  2. 根据应用程序所需的权限,编辑每个文件并添加其他规则,如下所示:

    • emr-containers-role-patch.yaml

      - apiGroups: - "" resources: - persistentvolumeclaims verbs: - list - create - delete
    • driver-role-patch.yaml

      - apiGroups: - "" resources: - persistentvolumeclaims verbs: - list - create - delete - apiGroups: - "" resources: - services verbs: - get - list - describe - create - delete - watch
    • client-role-patch.yaml

      - apiGroups: - "" resources: - persistentvolumeclaims verbs: - list - create - delete
  3. 删除以下属性及其值。这是应用更新所必需的。

    • creationTimestamp

    • resourceVersion

    • uid

  4. 最后,运行补丁:

    kubectl apply -f emr-containers-role-patch.yaml kubectl apply -f driver-role-patch.yaml kubectl apply -f client-role-patch.yaml