AWS::EMR::Cluster - AWS CloudFormation
AWS 文档中描述的 AWS 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅中国的 AWS 服务入门

AWS::EMR::Cluster

AWS::EMR::Cluster 资源指定 Amazon EMR 集群。此集群是 Amazon EC2 实例的集合,这些实例运行开源大数据框架和应用程序来处理和分析大量数据。有关更多信息,请参阅 Amazon EMR 管理指南

语法

要在 AWS CloudFormation 模板中声明此实体,请使用以下语法:

JSON

{ "Type" : "AWS::EMR::Cluster", "Properties" : { "AdditionalInfo" : Json, "Applications" : [ Application, ... ], "AutoScalingRole" : String, "BootstrapActions" : [ BootstrapActionConfig, ... ], "Configurations" : [ Configuration, ... ], "CustomAmiId" : String, "EbsRootVolumeSize" : Integer, "Instances" : JobFlowInstancesConfig, "JobFlowRole" : String, "KerberosAttributes" : KerberosAttributes, "LogUri" : String, "Name" : String, "ReleaseLabel" : String, "ScaleDownBehavior" : String, "SecurityConfiguration" : String, "ServiceRole" : String, "Steps" : [ StepConfig, ... ], "Tags" : [ Tag, ... ], "VisibleToAllUsers" : Boolean } }

属性

AdditionalInfo

一个用于选择其他功能的 JSON 字符串。

必需:否

类型:Json

最低0

最高10280

模式[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

Applications

要在此集群上安装的应用程序,例如 Spark、Flink、Oozie、Zeppelin 等。

必需:否

类型应用程序的列表

Update requires: Replacement

AutoScalingRole

自动扩展策略的 IAM 角色。默认角色是 EMR_AutoScaling_DefaultRole。IAM 角色提供自动扩展功能在实例组中启动和终止 EC2 实例所需的权限。

必需:否

类型:字符串

最低0

最高10280

模式[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

BootstrapActions

在集群节点上启动 Hadoop 之前要运行的引导操作的列表。

必需:否

类型BootstrapActionConfig 的列表

Update requires: Replacement

Configurations

仅适用于 Amazon EMR 版本 4.x 及更高版本。为 EMR 集群提供的配置的列表。

必需:否

类型配置的列表

Update requires: Replacement

CustomAmiId

仅在 Amazon EMR 5.7.0 版及更高版本中可用。如果集群使用自定义 AMI,则为由 Amazon EBS 支持的自定义 Linux AMI 的 ID。

必需:否

类型:字符串

最低0

最高256

模式[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

EbsRootVolumeSize

用于每个 EC2 实例的 Linux AMI 的 EBS 根设备卷的大小(以 GiB 为单位)。在 Amazon EMR 4.x 版及更高版本中可用。

必需:否

类型:整数

Update requires: Replacement

Instances

Amazon EC2 实例的数量和类型规范。

必需:是

类型JobFlowInstancesConfig

Update requires: Some interruptions

JobFlowRole

也称为实例配置文件和 EC2 角色。EMR 集群的 IAM 角色。集群中的 EC2 实例将代入此角色。默认角色是 EMR_EC2_DefaultRole。要使用默认角色,您必须已使用 CLI 或控制台创建它。

必需:是

类型:字符串

最低0

最高10280

模式[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

KerberosAttributes

使用安全配置启用 Kerberos 身份验证时的 Kerberos 配置的属性。有关更多信息,请参阅 EMR 管理指南 中的使用 Kerberos 身份验证

必需:否

类型KerberosAttributes

Update requires: Replacement

LogUri

用于存储此集群的日志的Amazon S3 位置的路径。

必需:否

类型:字符串

Update requires: Replacement

Name

集群的名称。

必需:是

类型:字符串

Update requires: Replacement

ReleaseLabel

Amazon EMR 发布版标签,它可确定集群上安装的开源应用程序包的版本。版本标签的格式为 emr-x.x.x,其中,x.x.x 为 Amazon EMR 发布版,例如 emr-5.14.0。有关 Amazon EMR 发布版和包含的应用程序版本和功能的更多信息,请参阅https://docs.amazonaws.cn/emr/latest/ReleaseGuide/。发布版标签仅适用于 Amazon EMR 发布版 4.0 及更高版本。早期版本使用 AmiVersion

必需:否

类型:字符串

Update requires: Replacement

ScaleDownBehavior

执行自动缩减活动或调整实例组大小时,各个 Amazon EC2 实例的终止方式。TERMINATE_AT_INSTANCE_HOUR 指示 Amazon EMR 在实例小时边界终止节点,无论何时提交终止实例的请求。此选项仅适用于 Amazon EMR 5.1.0 及更高版本,并且是使用该版本创建的集群的默认选项。TERMINATE_AT_TASK_COMPLETION 指示 Amazon EMR 在终止 Amazon EC2 实例之前将节点中的任务列入黑名单并排除任务,而不管实例小时边界如何。对于任一行为,Amazon EMR 都会先删除最不活跃的节点,并阻止实例终止(如果它可能导致 HDFS 损坏)。TERMINATE_AT_TASK_COMPLETION 仅在 Amazon EMR 4.1.0 版及更高版本中可用,并且是 5.1.0 版之前的 Amazon EMR 版本的默认值。

必需:否

类型:字符串

允许的值TERMINATE_AT_INSTANCE_HOUR | TERMINATE_AT_TASK_COMPLETION

Update requires: Replacement

SecurityConfiguration

应用于集群的安全配置的名称。

必需:否

类型:字符串

最低0

最高10280

模式[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*

Update requires: Replacement

ServiceRole

将由 Amazon EMR 服务代入以将代表您访问 AWS 资源的 IAM 角色。

必需:是

类型:字符串

Update requires: Replacement

Steps

要运行的步骤的列表。

必需:否

类型StepConfig 的列表

Update requires: Replacement

Tags

与集群关联的标签的列表。

必需:否

类型Tag 的列表

Update requires: No interruption

VisibleToAllUsers

指示集群是否对与其关联的 AWS 账户中的所有 IAM 用户可见。如果将此值设置为 true,则该 AWS 账户的所有 IAM 用户都可以查看和管理集群(如果他们具有正确的策略权限设置)。如果此值为 false,则仅创建了集群的 IAM 用户能够查看和管理它。可使用 SetVisibleToAllUsers 操作更改此值。

注意

直接通过 EMR 控制台或 API 创建群集时,默认情况下此值设置为 true。但是,对于CloudFormation 中的 AWS::EMR::Cluster 资源,默认值为 false

必需:否

类型:布尔值

Update requires: No interruption

返回值

Ref

在将此资源的逻辑 ID 传递给内部 Ref 函数时,Ref 返回集群 ID,例如 j-1ABCD123AB1A。

For more information about using the Ref function, see Ref.

Fn::GetAtt

Fn::GetAtt 内部函数返回此类型的一个指定属性的值。以下为可用属性和示例返回值。

有关使用 Fn::GetAtt 内部函数的更多信息,请参阅 Fn::GetAtt

MasterPublicDNS

主节点 (实例) 的公有 DNS 名称,如 ec2-12-123-123-123.us-west-2.compute.amazonaws.com

示例

创建集群示例。

使用 EC2 实例的自定义 AMI 创建集群

以下示例模板使用该集群中 EC2 实例的自定义 Amazon Linux AMI 指定集群。

JSON

{ "AWSTemplateFormatVersion": "2010-09-09", "Parameters" : { "CustomAmiId" : { "Type" : "String" }, "InstanceType" : { "Type" : "String" }, "ReleaseLabel" : { "Type" : "String" }, "SubnetId" : { "Type" : "String" }, "TerminationProtected" : { "Type" : "String", "Default" : "false" }, "ElasticMapReducePrincipal" : { "Type" : "String" }, "Ec2Principal" : { "Type" : "String" } }, "Resources": { "cluster": { "Type": "AWS::EMR::Cluster", "Properties": { "CustomAmiId" : {"Ref" : "CustomAmiId"}, "Instances": { "MasterInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnMaster" }, "CoreInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnCore" }, "TerminationProtected" : {"Ref" : "TerminationProtected"}, "Ec2SubnetId" : {"Ref" : "SubnetId"} }, "Name": "CFNtest", "JobFlowRole" : {"Ref": "emrEc2InstanceProfile"}, "ServiceRole" : {"Ref": "emrRole"}, "ReleaseLabel" : {"Ref" : "ReleaseLabel"}, "VisibleToAllUsers" : true, "Tags": [ { "Key": "key1", "Value": "value1" } ] } }, "emrRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": {"Ref" : "ElasticMapReducePrincipal"} }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"] } }, "emrEc2Role": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": {"Ref" : "Ec2Principal"} }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"] } }, "emrEc2InstanceProfile": { "Type": "AWS::IAM::InstanceProfile", "Properties": { "Path": "/", "Roles": [ { "Ref": "emrEc2Role" } ] } } } }

YAML

AWSTemplateFormatVersion: 2010-09-09 Parameters: CustomAmiId: Type: String InstanceType: Type: String ReleaseLabel: Type: String SubnetId: Type: String TerminationProtected: Type: String Default: 'false' ElasticMapReducePrincipal: Type: String Ec2Principal: Type: String Resources: cluster: Type: AWS::EMR::Cluster Properties: CustomAmiId: !Ref CustomAmiId Instances: MasterInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnMaster CoreInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnCore TerminationProtected: !Ref TerminationProtected Ec2SubnetId: !Ref SubnetId Name: CFNtest JobFlowRole: !Ref emrEc2InstanceProfile ServiceRole: !Ref emrRole ReleaseLabel: !Ref ReleaseLabel VisibleToAllUsers: true Tags: - Key: key1 Value: value1 emrRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: !Ref ElasticMapReducePrincipal Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole' emrEc2Role: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: !Ref Ec2Principal Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role' emrEc2InstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref emrEc2Role

创建集群并指定 EC2 实例的根卷大小

以下示例模板允许您指定集群实例的 EBS 根卷的大小。

JSON

{ "AWSTemplateFormatVersion": "2010-09-09", "Parameters" : { "InstanceType" : { "Type" : "String" }, "ReleaseLabel" : { "Type" : "String" }, "SubnetId" : { "Type" : "String" }, "TerminationProtected" : { "Type" : "String", "Default" : "false" }, "EbsRootVolumeSize" : { "Type" : "String" } }, "Resources": { "cluster": { "Type": "AWS::EMR::Cluster", "Properties": { "EbsRootVolumeSize" : {"Ref" : "EbsRootVolumeSize"}, "Instances": { "MasterInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnMaster" }, "CoreInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnCore" }, "TerminationProtected" : {"Ref" : "TerminationProtected"}, "Ec2SubnetId" : {"Ref" : "SubnetId"} }, "Name": "CFNtest", "JobFlowRole" : {"Ref": "emrEc2InstanceProfile"}, "ServiceRole" : {"Ref": "emrRole"}, "ReleaseLabel" : {"Ref" : "ReleaseLabel"}, "VisibleToAllUsers" : true, "Tags": [ { "Key": "key1", "Value": "value1" } ] } }, "emrRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "elasticmapreduce.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"] } }, "emrEc2Role": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"] } }, "emrEc2InstanceProfile": { "Type": "AWS::IAM::InstanceProfile", "Properties": { "Path": "/", "Roles": [ { "Ref": "emrEc2Role" } ] } } } }

YAML

AWSTemplateFormatVersion: 2010-09-09 Parameters: InstanceType: Type: String ReleaseLabel: Type: String SubnetId: Type: String TerminationProtected: Type: String Default: 'false' EbsRootVolumeSize: Type: String Resources: cluster: Type: AWS::EMR::Cluster Properties: EbsRootVolumeSize: !Ref EbsRootVolumeSize Instances: MasterInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnMaster CoreInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnCore TerminationProtected: !Ref TerminationProtected Ec2SubnetId: !Ref SubnetId Name: CFNtest JobFlowRole: !Ref emrEc2InstanceProfile ServiceRole: !Ref emrRole ReleaseLabel: !Ref ReleaseLabel VisibleToAllUsers: true Tags: - Key: key1 Value: value1 emrRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: elasticmapreduce.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole' emrEc2Role: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: ec2.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role' emrEc2InstanceProfile: Type: AWS::IAM::InstanceProfile Properties: Path: / Roles: - !Ref emrEc2Role

创建具有 Kerberos 身份验证的集群

以下示例模板允许您指定 EMR 集群的 Kerberos 身份验证配置。

JSON

{ "AWSTemplateFormatVersion": "2010-09-09", "Parameters" : { "CrossRealmTrustPrincipalPassword" : { "Type" : "String" }, "KdcAdminPassword" : { "Type" : "String" }, "Realm" : { "Type" : "String" }, "InstanceType" : { "Type" : "String" }, "ReleaseLabel" : { "Type" : "String" }, "SubnetId" : { "Type" : "String" } }, "Resources": { "cluster": { "Type": "AWS::EMR::Cluster", "Properties": { "Instances": { "MasterInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnMaster" }, "CoreInstanceGroup": { "InstanceCount": 1, "InstanceType": {"Ref" : "InstanceType"}, "Market": "ON_DEMAND", "Name": "cfnCore" }, "Ec2SubnetId" : {"Ref" : "SubnetId"} }, "Name": "CFNtest2", "JobFlowRole" : {"Ref": "emrEc2InstanceProfile"}, "KerberosAttributes" : { "CrossRealmTrustPrincipalPassword" : "CfnIntegrationTest-1", "KdcAdminPassword" : "CfnIntegrationTest-1", "Realm": "EC2.INTERNAL" }, "ServiceRole" : {"Ref": "emrRole"}, "ReleaseLabel" : {"Ref" : "ReleaseLabel"}, "SecurityConfiguration" : {"Ref" : "securityConfiguration"}, "VisibleToAllUsers" : true, "Tags": [ { "Key": "key1", "Value": "value1" } ] } }, "key" : { "Type" : "AWS::KMS::Key", "Properties" : { "KeyPolicy" : { "Version": "2012-10-17", "Id": "key-default-1", "Statement": [ { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": { "Fn::GetAtt" : ["emrEc2Role", "Arn"]} }, "Action": "kms:*", "Resource": "*" }, { "Sid": "Enable IAM User Permissions", "Effect": "Allow", "Principal": { "AWS": { "Fn::Join" : ["" , ["arn:aws:iam::", {"Ref" : "AWS::AccountId"} ,":root" ]] } }, "Action": "kms:*", "Resource": "*" } ] } } }, "securityConfiguration": { "Type" : "AWS::EMR::SecurityConfiguration", "Properties" : { "SecurityConfiguration" : { "AuthenticationConfiguration": { "KerberosConfiguration": { "Provider": "ClusterDedicatedKdc", "ClusterDedicatedKdcConfiguration": { "TicketLifetimeInHours": 24, "CrossRealmTrustConfiguration": { "Realm": "AD.DOMAIN.COM", "Domain": "ad.domain.com", "AdminServer": "ad.domain.com", "KdcServer": "ad.domain.com" } } } } } } }, "emrRole": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "elasticmapreduce.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole"] } }, "emrEc2Role": { "Type": "AWS::IAM::Role", "Properties": { "AssumeRolePolicyDocument": { "Version": "2008-10-17", "Statement": [ { "Sid": "", "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" }, "Action": "sts:AssumeRole" } ] }, "Path": "/", "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role"] } }, "emrEc2InstanceProfile": { "Type": "AWS::IAM::InstanceProfile", "Properties": { "Path": "/", "Roles": [ { "Ref": "emrEc2Role" } ] } } }, "Outputs" : { "keyArn" : { "Value" : {"Fn::GetAtt" : ["key", "Arn"]} } } }

YAML

AWSTemplateFormatVersion: 2010-09-09 Parameters: CrossRealmTrustPrincipalPassword: Type: String KdcAdminPassword: Type: String Realm: Type: String InstanceType: Type: String ReleaseLabel: Type: String SubnetId: Type: String Resources: cluster: Type: 'AWS::EMR::Cluster' Properties: Instances: MasterInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnMaster CoreInstanceGroup: InstanceCount: 1 InstanceType: !Ref InstanceType Market: ON_DEMAND Name: cfnCore Ec2SubnetId: !Ref SubnetId Name: CFNtest2 JobFlowRole: !Ref emrEc2InstanceProfile KerberosAttributes: CrossRealmTrustPrincipalPassword: CfnIntegrationTest-1 KdcAdminPassword: CfnIntegrationTest-1 Realm: EC2.INTERNAL ServiceRole: !Ref emrRole ReleaseLabel: !Ref ReleaseLabel SecurityConfiguration: !Ref securityConfiguration VisibleToAllUsers: true Tags: - Key: key1 Value: value1 key: Type: 'AWS::KMS::Key' Properties: KeyPolicy: Version: 2012-10-17 Id: key-default-1 Statement: - Sid: Enable IAM User Permissions Effect: Allow Principal: AWS: !GetAtt - emrEc2Role - Arn Action: 'kms:*' Resource: '*' - Sid: Enable IAM User Permissions Effect: Allow Principal: AWS: !Join - '' - - 'arn:aws:iam::' - !Ref 'AWS::AccountId' - ':root' Action: 'kms:*' Resource: '*' securityConfiguration: Type: 'AWS::EMR::SecurityConfiguration' Properties: SecurityConfiguration: AuthenticationConfiguration: KerberosConfiguration: Provider: ClusterDedicatedKdc ClusterDedicatedKdcConfiguration: TicketLifetimeInHours: 24 CrossRealmTrustConfiguration: Realm: AD.DOMAIN.COM Domain: ad.domain.com AdminServer: ad.domain.com KdcServer: ad.domain.com emrRole: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: elasticmapreduce.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceRole' emrEc2Role: Type: 'AWS::IAM::Role' Properties: AssumeRolePolicyDocument: Version: 2008-10-17 Statement: - Sid: '' Effect: Allow Principal: Service: ec2.amazonaws.com Action: 'sts:AssumeRole' Path: / ManagedPolicyArns: - 'arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role' emrEc2InstanceProfile: Type: 'AWS::IAM::InstanceProfile' Properties: Path: / Roles: - !Ref emrEc2Role Outputs: keyArn: Value: !GetAtt - key - Arn