Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Using Trusted Identity Propagation with Amazon SageMaker Unified Studio

This tutorial shows how to set up an EMR Serverless application with Trusted Identity Propagation (TIP). You then query a sample table using identity-based access controls through Amazon SageMaker Unified Studio. With this setup, EMR Serverless runs each user's Spark queries with their own IAM Identity Center identity, and Lake Formation enforces per-user permissions on Amazon Glue Data Catalog tables.

Prerequisites and setup

Complete the following steps to set up all required resources before querying data with trusted identity propagation.

Step 1: Set up IAM Identity Center

Step 2: Configure Lake Formation with IAM Identity Center

Step 3: Create the TIP-enabled EMR Serverless application

Ensure your IAM role has the required IAM Identity Center permissions to create the application:

"sso:DescribeInstance",
"sso:CreateApplication", 
"sso:DeleteApplication",
"sso:PutApplicationAuthenticationMethod",
"sso:PutApplicationAssignmentConfiguration",  
"sso:PutApplicationGrant", 
"sso:PutApplicationAccessScope"

Create the application with trusted identity propagation, Apache Livy endpoint enabled, and the Amazon Glue Data Catalog configured as the metastore:

aws emr-serverless create-application \
    --release-label emr-7.8.0 \
    --type "SPARK" \
    --identity-center-configuration '{"identityCenterInstanceArn": "arn:aws:sso:::instance/ssoins-instance-id"}' \
    --interactive-configuration '{"livyEndpointEnabled":true}' \
    --runtime-configuration '[{"classification": "spark-defaults", "properties": {"spark.sql.catalogImplementation": "hive", "spark.hadoop.hive.metastore.client.factory.class": "com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory"}}]'

import boto3

session = boto3.Session(profile_name='your-profile', region_name='your-region')
client = session.client('emr-serverless')

response = client.create_application(
    name='tip-tutorial-app',
    releaseLabel='emr-7.8.0',
    type='SPARK',
    identityCenterConfiguration={
        'identityCenterInstanceArn': 'arn:aws:sso:::instance/ssoins-instance-id'
    },
    interactiveConfiguration={
        'livyEndpointEnabled': True
    },
    runtimeConfiguration=[
        {
            'classification': 'spark-defaults',
            'properties': {
                'spark.sql.catalogImplementation': 'hive',
                'spark.hadoop.hive.metastore.client.factory.class': 'com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory'
            }
        }
    ]
)
print(response['applicationId'])

If you don't have the required IAM Identity Center permissions, create the application without the --identity-center-configuration parameter and ask your IAM Identity Center administrator to enable TIP later using the update-application API:

aws emr-serverless update-application \
    --application-id application-id \
    --identity-center-configuration '{"identityCenterInstanceArn": "arn:aws:sso:::instance/ssoins-instance-id"}'

Step 4: Configure the job execution role

The job execution role's trust policy must include sts:SetContext to allow EMR Serverless to enhance credentials with the user's identity:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "emr-serverless.amazonaws.com"
            },
            "Action": ["sts:AssumeRole", "sts:SetContext"]
        }
    ]
}

Create the role:

aws iam create-role \
    --role-name emr-serverless-tip-job-role \
    --assume-role-policy-document file://trust-policy.json \
    --description "EMR Serverless job execution role with TIP support"

Attach permissions for the downstream services the job will access. The role needs Lake Formation, Amazon Glue, Amazon S3, and Amazon CloudWatch Logs permissions:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LakeFormationAndGlueAccess",
            "Effect": "Allow",
            "Action": [
                "lakeformation:GetDataAccess",
                "glue:GetTable", "glue:GetTables",
                "glue:GetDatabase", "glue:GetDatabases",
                "glue:GetPartitions"
            ],
            "Resource": "*"
        },
        {
            "Sid": "S3DataAccess",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::your-data-bucket",
                "arn:aws:s3:::your-data-bucket/*"
            ]
        },
        {
            "Sid": "CloudWatchLogsAccess",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents",
                "logs:DescribeLogGroups",
                "logs:DescribeLogStreams"
            ],
            "Resource": "arn:aws:logs:us-east-1:111122223333:*"
        }
    ]
}

Attach the policy to the role:

aws iam put-role-policy \
    --role-name emr-serverless-tip-job-role \
    --policy-name tip-lakeformation-access \
    --policy-document file://permissions-policy.json

Grant Lake Formation permissions to the job execution role:

The job execution role also requires Lake Formation permissions on the database and table. This is in addition to the grants given to the IAM Identity Center user.

# Grant DESCRIBE on the database to the job role
aws lakeformation grant-permissions \
    --principal '{"DataLakePrincipalIdentifier": "arn:aws:iam::account-id:role/emr-serverless-tip-job-role"}' \
    --permissions '["DESCRIBE"]' \
    --resource '{"Database": {"DatabaseName": "tip_tutorial_db", "CatalogId": "account-id"}}'

# Grant SELECT and DESCRIBE on the table to the job role
aws lakeformation grant-permissions \
    --principal '{"DataLakePrincipalIdentifier": "arn:aws:iam::account-id:role/emr-serverless-tip-job-role"}' \
    --permissions '["SELECT","DESCRIBE"]' \
    --resource '{"Table": {"DatabaseName": "tip_tutorial_db", "Name": "employees", "CatalogId": "account-id"}}'

For additional downstream service configurations, see Using Lake Formation with EMR Serverless and Using Amazon S3 Access Grants with EMR Serverless.

Query data with Trusted Identity Propagation through Amazon SageMaker Unified Studio

After you complete the setup, you can query data using your TIP-enabled EMR Serverless application as compute in a Amazon SageMaker Unified Studio project. EMR Serverless runs each user's queries with their own IAM Identity Center identity, and Lake Formation enforces per-user permissions.

Prerequisites

Step 5: Create a Amazon SageMaker Unified Studio project with TIP enabled

Step 6: Add the EMR Serverless application as compute

Amazon SageMaker Unified Studio project compute configuration does not currently support adding an existing EMR Serverless application through the console. Use the Amazon CLI to create the connection:

aws datazone create-connection \
  --domain-identifier domain-id \
  --name "my-emr-serverless-connection" \
  --environment-identifier environment-id \
  --props '{
    "sparkEmrProperties": {
      "computeArn": "arn:aws:emr-serverless:region:account-id:/applications/application-id",
      "runtimeRole": "arn:aws:iam::account-id:role/emr-serverless-tip-job-role"
    }
  }'

Replace the following values:

import boto3

session = boto3.Session(profile_name='your-profile', region_name='your-region')
client = session.client('datazone')

client.create_project_membership(
    domainIdentifier='domain-id',
    projectIdentifier='project-id',
    designation='PROJECT_CONTRIBUTOR',
    member={'userIdentifier': 'arn:aws:iam::account-id:role/your-role'}
)

Configure the project's DataZone user role for EMR Serverless access.

The DataZone user role for the project environment must have permissions to interact with the EMR Serverless application. Add the following inline policy to the role (the role name follows the pattern datazone_usr_role_<project-id>_<environment-id>):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "emr-serverless:GetApplication",
                "emr-serverless:StartApplication",
                "emr-serverless:StopApplication",
                "emr-serverless:ListApplications",
                "emr-serverless:AccessLivyEndpoints",
                "emr-serverless:StartJobRun",
                "emr-serverless:GetJobRun",
                "emr-serverless:ListJobRuns",
                "emr-serverless:CancelJobRun",
                "emr-serverless:GetDashboardForJobRun"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::111122223333:role/emr-serverless-tip-job-role",
            "Condition": {
                "StringLike": {
                    "iam:PassedToService": "emr-serverless.amazonaws.com"
                }
            }
        }
    ]
}

Grant Lake Formation permissions to the DataZone user role:

# Grant DESCRIBE on the database
aws lakeformation grant-permissions \
    --principal '{"DataLakePrincipalIdentifier": "arn:aws:iam::account-id:role/datazone_usr_role_project-id_environment-id"}' \
    --permissions '["DESCRIBE"]' \
    --resource '{"Database": {"DatabaseName": "tip_tutorial_db", "CatalogId": "account-id"}}'

# Grant SELECT and DESCRIBE on the table
aws lakeformation grant-permissions \
    --principal '{"DataLakePrincipalIdentifier": "arn:aws:iam::account-id:role/datazone_usr_role_project-id_environment-id"}' \
    --permissions '["SELECT","DESCRIBE"]' \
    --resource '{"Table": {"DatabaseName": "tip_tutorial_db", "Name": "employees", "CatalogId": "account-id"}}'

Step 7: Query a sample table

Troubleshooting

Clean up resources

To avoid ongoing charges, delete the resources you created in this tutorial:

Next steps