Configure the discoverability of Amazon EMR clusters - Amazon SageMaker
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Configure the discoverability of Amazon EMR clusters

Administrators can configure the discoverability of running Amazon EMR clusters from Studio or Studio Classic. The clusters can be deployed in the same Amazon account as Studio (choose the Single account tab) or in separate accounts (choose the Cross account tab).

Single account

If your Amazon EMR clusters and Studio or Studio Classic are deployed in the same Amazon account, attach the following permissions to the SageMaker execution role accessing your cluster.

Note

What execution role should you consider?

The execution role attached to your domain might be different from the role set on your user profile or the space used by an application.

For consistent access to Amazon EMR templates and clusters across applications (such as the Studio UI, JupyterLab, and Studio Classic), grant the same subset of permissions to all roles. The permissions should allow discovering and provisioning Amazon EMR clusters.

  1. Find the execution role of your domain, user profile, or space. For information on how to retrieve the execution role, see Get execution role.

  2. Open the IAM console at https://console.amazonaws.cn/sagemaker/.

  3. Choose Roles and then search for the role you created by typing in your role name in the Search field.

  4. Follow the link to your role.

  5. Choose Add permissions and then Create inline policy.

  6. In the JSON tab, add the following JSON policy with the permissions:

    • AllowSagemakerProjectManagement enables the creation of . In Studio or Studio Classic, access to the Amazon Service Catalog is granted through .

    • AllowClusterDetailsDiscovery and AllowClusterDiscovery allow the discovery and connection to Amazon EMR clusters.

    • AllowPresignedUrl allows the creation of pre-signed URLs to access Spark UI.

    The IAM policy defined in the provided JSON grants those permissions. Replace studio-region and studio-account with your actual region and Amazon account ID values before copying the list of statements to the inline policy of your role.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPresignedUrl",
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:CreatePersistentAppUI",
                "elasticmapreduce:DescribePersistentAppUI",
                "elasticmapreduce:GetPersistentAppUIPresignedURL",
                "elasticmapreduce:GetOnClusterAppUIPresignedURL"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:studio-region:studio-account:cluster/*"
            ]
        },
        {
            "Sid": "AllowClusterDetailsDiscovery",
            "Effect": "Allow",
            "Action": [
               "elasticmapreduce:DescribeCluster",
               "elasticmapreduce:ListInstances",
               "elasticmapreduce:ListInstanceGroups",
               "elasticmapreduce:DescribeSecurityConfiguration"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:studio-region:studio-account:cluster/*"
            ]
        },
        {
            "Sid": "AllowClusterDiscovery",
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:ListClusters"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowSagemakerProjectManagement",
            "Effect": "Allow",
            "Action": [
                "sagemaker:CreateProject",
                "sagemaker:DeleteProject"
            ],
            "Resource": "arn:aws:sagemaker:studio-region:studio-account:project/*"
        }
    ]
}

  7. Name your policy and choose Create policy.

Cross account

If your Amazon EMR clusters and Studio or Studio Classic are deployed in separate Amazon accounts, you configure the permissions on both accounts.

  • On the account where Amazon EMR is deployed, also referred to as the trusting account, create a custom IAM role named ASSUMABLE-ROLE with the following configuration:

    • Permissions: Grant the necessary permissions to ASSUMABLE-ROLE to allow accessing Amazon EMR resources.

    • Trust relationship: Configure the trust policy for ASSUMABLE-ROLE to allow assuming the role from the Studio account that requires access.

    By assuming the role, Studio or Studio Classic can gain temporary access to the permissions it needs in Amazon EMR.

    Create a new policy for the role:

    1. Open the IAM console at https://console.amazonaws.cn/sagemaker/.

    2. In the left menu, choose Policies and then Create policy.

    3. In the JSON tab, add the following JSON policy with the permissions:

      • AllowClusterDetailsDiscovery and AllowClusterDiscovery to allow the discovery and connection to Amazon EMR clusters.

      • AllowPresignedUrl to allow the creation of pre-signed URLs to access Spark UI.

      Replace emr-region and emr-account with your actual region and Amazon account ID values before copying the JSON to your policy.

      {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowPresignedUrl",
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:DescribeCluster",
                "elasticmapreduce:ListInstanceGroups",
                "elasticmapreduce:CreatePersistentAppUI",
                "elasticmapreduce:DescribePersistentAppUI",
                "elasticmapreduce:GetPersistentAppUIPresignedURL",
                "elasticmapreduce:GetOnClusterAppUIPresignedURL"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:emr-region:emr-account:cluster/*"
            ]
        },
        {
            "Sid": "AllowClusterDetailsDiscovery",
            "Effect": "Allow",
            "Action": [
               "elasticmapreduce:DescribeCluster",
               "elasticmapreduce:ListInstances",
               "elasticmapreduce:ListInstanceGroups",
               "elasticmapreduce:DescribeSecurityConfiguration"
            ],
            "Resource": [
                "arn:aws:elasticmapreduce:emr-region:emr-account:cluster/*"
            ]
        },
        {
            "Sid": "AllowClusterDiscovery",
            "Effect": "Allow",
            "Action": [
                "elasticmapreduce:ListClusters"
            ],
            "Resource": "*"
        }
    ]
}

    4. Name your policy and choose Create policy.

  • Create a custom IAM role named ASSUMABLE-ROLE, and then attach your new policy to the role.

    1. In the IAM console, choose Roles in the left menu, and then Create role.

    2. For Trusted entity type, choose Amazon account and then Next.

    3. Select the permission you just created and then choose Next.

    4. Name your role ASSUMABLE-ROLE and then include the following trust relationship in Trust policy. This grants the account where Studio is deployed (the trusted account) the permission to assume this role.

      Replace studio-account with its actual Amazon account ID. Choose Create role.

      {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::studio-account:root"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

    5. Choose Create role.

  • On the account where Studio or Studio Classic is deployed, also referred to as the trusted account, update the SageMaker execution role accessing your cluster with the following trust policy.

    The trust policy allows the execution role to assume the ASSUMABLE-ROLE in the trusting account to securely list resources in the trusting account by assuming the role.

    Note

    What execution role should you consider?

    The execution role attached to your domain might be different from the role set on your user profile or the space used by an application.

    For consistent access to Amazon EMR templates and clusters across applications (such as the Studio UI and Studio Classic), grant the same subset of permissions to all roles. The permissions should allow discovering and provisioning Amazon EMR clusters.

    1. Find the execution role of your domain, user profile, or space. For information on how to retrieve the execution role, see Get execution role.

    2. Open the IAM console at https://console.amazonaws.cn/sagemaker/.

    3. Choose Roles and then search for the role you created by typing in your role name in the Search field.

    4. Follow the link to your role.

    5. Choose the Trust relationships and then Edit trust policy. Replace emr-account with your actual Amazon EMR account ID value before copying the JSON to your policy.

      {
  "Version": "2012-10-17",
  "Statement": [
  { 
            "Sid": "AllowRoleAssumptionForCrossAccountDiscovery", 
            "Effect": "Allow", 
            "Action": "sts:AssumeRole", 
            "Resource":  ["arn:aws:iam::emr-account:role/ASSUMABLE-ROLE" ]
  }]
}

    6. Choose Update policy.

  • Last, see Additional configuration for cross-account use cases to learn about how to provide the ARN of the ASSUMABLE-ROLE to your Studio execution role. The ARN is loaded by the Jupyter server at launch. The execution role used by Studio assumes that cross-account role to discover Amazon EMR clusters in the trusting account.

Visit List Amazon EMR clusters from Studio or Studio Classic to learn about how to discover and connect to Amazon EMR clusters from Studio or Studio Classic notebooks.