Sharing a data lake using Lake Formation tag-based access control and named resources

Configure Lake Formation Data Catalog settings in the producer account

Lake Formation provides its own permission management model. To maintain backward compatibility with the IAM permission model, the Super permission is granted to the group IAMAllowedPrincipals on all existing Amazon Glue Data Catalog resources by default. Also, Use only IAM access control settings are enabled for new Data Catalog resources. This tutorial uses fine grained access control using Lake Formation permissions and use IAM policies for coarse grained access control. See Methods for fine-grained access control for details. Therefore, before you use an Amazon CloudFormation template for a quick setup, you need to change Lake Formation Data Catalog settings in the producer account.

To configure Lake Formation Data Catalog settings in the producer account, complete the following steps:

1. Sign into the Amazon Web Services Management Console using the producer account as an admin user, or as a user with Lake Formation PutDataLakeSettings API permission.

2. On the Lake Formation console, in the navigation pane, under Data Catalog, choose Settings.

3. Deselect Use only IAM access control for new databases and Use only IAM access control for new tables in new databases

   Choose Save.

   

   Additionally, you can remove CREATE_DATABASE permissions for IAMAllowedPrincipals under Administrative roles and tasks, Database creators. Only then, you can govern who can create a new database through Lake Formation permissions.

Create your resources in the producer account

1. Sign into the Amazon CloudFormation console at https://console.amazonaws.cn/cloudformation in the US East (N. Virginia) region.

2. Choose Launch Stack.

3. Choose Next.

4. For Stack name, enter a stack name, such as stack-producer.

5. In the User Configuration section, enter user name and password for ProducerDatalakeAdminUserName and ProducerDatalakeAdminUserPassword.

6. For DataLakeBucketName, enter the name of your data lake bucket. This name needs to be globally unique.

7. For DatabaseName and TableName, leave the default values.

8. Choose Next.

9. On the next page, choose Next.

10. Review the details on the final page and select I acknowledge that Amazon CloudFormation might create IAM resources.

11. Choose Create.

    The stack creation can take up to one minute.

Create your resources in the consumer account

1. Sign into the Amazon CloudFormation console at https://console.amazonaws.cn/cloudformation in the US East (N. Virginia) region.

2. Choose Launch Stack.

3. Choose Next.

4. For Stack name, enter a stack name, such as stack-consumer.

5. In the User Configuration section, enter user name and password for ConsumerDatalakeAdminUserName and ConsumerDatalakeAdminUserPassword.

6. For DataAnalystUserName and DataAnalystUserPassword, enter the user name and password you want for the data analyst IAM user.

7. For DataLakeBucketName, enter the name of your data lake bucket. This name needs to be globally unique.

8. For DatabaseName, leave the default values.

9. For AthenaQueryResultS3BucketName, enter the name of the Amazon S3 bucket that stores Amazon Athena query results. If you don’t have one, create an Amazon S3 bucket.

10. Choose Next.

11. On the next page, choose Next.

12. Review the details on the final page and select I acknowledge that Amazon CloudFormation might create IAM resources.

13. Choose Create.

    The stack creation can take up to one minutes.

Complete tag-based access control cross-account data sharing prerequisites

• For more information on cross-account data sharing requirements, see the Prerequisites section in the Cross-account data sharing chapter.

  To share Data Catalog resources with version 3 or above of the Cross account version settings, the grantor requires to have the IAM permissions defined in the Amazon managed policy AWSLakeFormationCrossAccountManager in your account.

  If you are using version 1 or version 2 of the Cross account version settings, before you can use the tag-based access control method to grant cross-account access to resources, you must add the following JSON permissions object to the Data Catalog resource policy in the producer account. This gives the consumer account permission to access the Data Catalog when glue:EvaluatedByLakeFormationTags is true. Also, this condition becomes true for resources on which you granted permission using Lake Formation permission tags to the consumer’s account. This policy is required for every Amazon Web Services account to which you are granting permissions.

  The following policy must be within a Statement element. We discuss the full IAM policy in the next section.

  {
      "Effect": "Allow",
      "Action": [
          "glue:*"
      ],
      "Principal": {
          "AWS": [
              "consumer-account-id"
          ]
      },
      "Resource": [
          "arn:aws:glue:region:account-id:table/*",
          "arn:aws:glue:region:account-id:database/*",
          "arn:aws:glue:region:account-id:catalog"
      ],
      "Condition": {
          "Bool": {
              "glue:EvaluatedByLakeFormationTags": true
          }
      }
  }
          

Complete named resource method cross-account sharing prerequisites

1. If there is no Data Catalog resource policy in your account, the Lake Formation cross-account grants that you make proceed as usual. However, if a Data Catalog resource policy exists, you must add the following statement to it to permit your cross-account grants to succeed if they’re made with the named resource method. If you plan to use only the named resource method, or only the tag-based access control method, you can skip this step. In this tutorial, we evaluate both methods, and we need to add the following policy.

   The following policy must be within a Statement element. We discuss the full IAM policy in the next section.

   {
             "Effect": "Allow",
             "Action": [
             "glue:ShareResource"
             ],
             "Principal": {
               "Service":"ram.amazonaws.com"
             },
             "Resource": [
                 "arn:aws:glue:region:account-id:table/*/*",
                 "arn:aws:glue:region:account-id:database/*",
                 "arn:aws:glue:region:account-id:catalog"
             ]
   }
           

2. Next, add the Amazon Glue Data Catalog resource policy using the Amazon Command Line Interface (Amazon CLI).

   If you grant cross-account permissions by using both the tag-based access control method and named resource method, you must set the EnableHybrid argument to ‘true’ when adding the preceding policies. Because this option is not currently supported on the console, and you must use the glue:PutResourcePolicy API and Amazon CLI.

   First, create a policy document (such as policy.json) and add the preceding two policies. Replace consumer-account-id with the account ID of the Amazon Web Services account receiving the grant, region with the Region of the Data Catalog containing the databases and tables that you are granting permissions on, and account-id with the producer Amazon Web Services account ID.

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Principal": {
                   "Service": "ram.amazonaws.com"
               },
               "Action": "glue:ShareResource",
               "Resource": [
                   "arn:aws:glue:region:account-id:table/*/*",
                   "arn:aws:glue:region:account-id:database/*",
                   "arn:aws:glue:region:account-id:catalog"
               ]
           },
           {
               "Effect": "Allow",
               "Principal": {
                   "AWS": "region:account-id"
               },
               "Action": "glue:*",
               "Resource": [
                   "arn:aws:glue:region:account-id:table/*/*",
                   "arn:aws:glue:region:account-id:database/*",
                   "arn:aws:glue:region:account-id:catalog"
               ],
               "Condition": {
                   "Bool": {
                       "glue:EvaluatedByLakeFormationTags": "true"
                   }
               }
           }
       ]
   }        

   Enter the following Amazon CLI command. Replace glue-resource-policy with the correct values (such as file://policy.json).

   aws glue put-resource-policy --policy-in-json glue-resource-policy --enable-hybrid TRUE

   For more information, see put-resource-policy.

Define an LF-Tag

1. Sign into the producer account as the data lake administrator at https://console.amazonaws.cn/lakeformation/. Use the producer account number, IAM user name (the default is DatalakeAdminProducer), and password that you specified during Amazon CloudFormation stack creation.

2. On the Lake Formation console (https://console.amazonaws.cn/lakeformation/), in the navigation pane, under Permissions, choose LF-Tags and Permissions.

3. Choose Add LF-Tag.

Assign the LF-Tag to the target resource

Assign the LF-Tag to the target resource and grant data permissions to another account

As a data lake administrator, you can attach tags to resources. If you plan to use a separate role, you may have to grant describe and attach permissions to the separate role.

1. In the navigation pane, under Data Catalog, select Databases.

2. Select the target database (lakeformation_tutorial_cross_account_database_tbac) and on the Actions menu, choose Edit LF-Tags.

   For this tutorial, you assign an LF-Tag to a database, but you can also assign LF-Tags to tables and columns.

3. Choose Assign new LF-Tag.

4. Add the key Confidentiality and value public.

5. Choose Save.

Grant LF-Tag permission to the consumer account

Still in the producer account, grant permissions to the consumer account to access the LF-Tag.

1. In the navigation pane, under Permissions, choose LF-Tags and permissions.

2. Choose the LF-Tags tab, and choose the key and values of the LF-Tag that is being shared with the consumer account (key Confidentiality and value public).

3. Choose Grant permissions.

4. For Permission type, choose LF-Tag key-value pair permissions.

5. For Principals, choose External accounts.

6. Enter the target Amazon Web Services account ID.

   Amazon Web Services accounts within the same organization appear automatically. Otherwise, you have to manually enter the Amazon Web Services account ID.

7. Under Permissions, select Describe.

   This is the permissions given to the consumer account. Grantable permissions are permissions that the consumer account can grant to other principals.

8. Choose Grant.

   At this point, the consumer data lake administrator should be able to find the policy tag being shared via the consumer account Lake Formation console, under Permissions, LF-Tags and permissions.

Grant data permission to the consumer account

We will now provide data access to the consumer account by specifying an LF-Tag expression and granting the consumer account access to any table or database that matches the expression..

1. In the navigation pane, under Permissions,Data lake permissions, choose Grant.

2. For Principals, choose External accounts, and enter the target Amazon Web Services account ID.

3. For LF-Tags or catalog resources, choose the key and values of the LF-Tag that is being shared with the consumer account (key Confidentiality and value public).

4. For Permissions, under Resources matched by LF-Tags (recommended) choose Add LF-Tag.

5. Select the key and value of the tag that is being shared with the consumer account (key Confidentiality and value public).

6. For Database permissions, select Describe under Database permissions to grant access permissions at the database level.

7. The consumer data lake administrator should be able to find the policy tag being shared via the consumer account on the Lake Formation console at https://console.amazonaws.cn/lakeformation/, under Permissions, Administrative roles and tasks, LF-Tags.

8. Select Describe under Grantable permissions so the consumer account can grant database-level permissions to its users.

9. For Table and column permissions, select Select and Describe under Table permissions.

10. Select Select and Describe under Grantable permissions.

11. Choose Grant.

Revoke permission for IAMAllowedPrincipals on the database, tables, and columns (Optional).

At the very beginning of this tutorial, you changed the Lake Formation Data Catalog settings. If you skipped that part, this step is required. If you changed your Lake Formation Data Catalog settings, you can skip this step.

In this step, we need to revoke the default Super permission from IAMAllowedPrincipals on the database or table. See Step 4: Switch your data stores to the Lake Formation permissions model for details.

Before revoking permission for IAMAllowedPrincipals, make sure that you granted existing IAM principals with necessary permission through Lake Formation. This includes three steps:

1. Add IAM permission to the target IAM user or role with the Lake Formation GetDataAccess action (with IAM policy).

2. Grant the target IAM user or role with Lake Formation data permissions (alter, select, and so on).

3. Then, revoke permissions for IAMAllowedPrincipals. Otherwise, after revoking permissions for IAMAllowedPrincipals, existing IAM principals may no longer be able to access the target database or Data Catalog.

   Revoking Super permission for IAMAllowedPrincipals is required when you want to apply the Lake Formation permission model (instead of the IAM policy model) to manage user access within a single account or among multiple accounts using the Lake Formation permission model. You do not have to revoke permission of IAMAllowedPrincipals for other tables where you want to keep the traditional IAM policy model.

   At this point, the consumer account data lake administrator should be able to find the database and table being shared via the consumer account on the Lake Formation console at https://console.amazonaws.cn/lakeformation/, under Data Catalog, databases. If not, confirm if the following are properly configured:

   1. The correct policy tag and values are assigned to the target databases and tables.

   2. The correct tag permission and data permission are assigned to the consumer account.

   3. Revoke the default super permission from IAMAllowedPrincipals on the database or table.

Create a resource link to the shared table

When a resource is shared between accounts, and the shared resources are not put in the consumer accounts’ Data Catalog. To make them available, and query the underlying data of a shared table using services like Athena, we need to create a resource link to the shared table. A resource link is a Data Catalog object that is a link to a local or shared database or table. For details, see Creating resource links. By creating a resource link, you can:

To create a resource link, complete the following steps:

1. If you are signed into your consumer account, sign out.

2. Sign in as the consumer account data lake administrator. Use the consumer account ID, IAM user name (default DatalakeAdminConsumer) and password that you specified during Amazon CloudFormation stack creation.

3. On the Lake Formation console (https://console.amazonaws.cn/lakeformation/), in the navigation pane, under Data Catalog, Databases, choose the shared database lakeformation_tutorial_cross_account_database_tbac.

   If you don’t see the database, revisit the previous steps to see if everything is properly configured.

4. Choose View Tables.

5. Choose the shared table amazon_reviews_table_tbac.

6. On the Actions menu, choose Create resource link.

7. For Resource link name, enter a name (for this tutorial, amazon_reviews_table_tbac_resource_link).

8. Under Database, select the database that the resource link is created in (for this post, the Amazon CloudFormationn stack created the database lakeformation_tutorial_cross_account_database_consumer).

9. Choose Create.

   The resource link appears under Data catalog, Tables.

Create an LF-tag and assign it to the target database

Lake Formation tags reside in the same Data Catalog as the resources. This means that tags created in the producer account are not available to use when granting access to the resource links in the consumer account. You need to create a separate set of LF-tags in the consumer account to use LF tag-based access control when sharing the resource links in the consumer account.

1. Define the LF-tag in the consumer account. For this tutorial, we use key Division and values sales, marketing, and analyst.

2. Assign the LF-tag key Division and value analyst to the database lakeformation_tutorial_cross_account_database_consumer, where the resource link is created.

Grant LF-tag data permission to the consumer

As a final step, grant LF-tag data permission to the consumer.

1. In the navigation pane, under Permissions, Data lake permissions, choose Grant.

2. For Principals, choose IAM users and roles, and choose the user DataAnalyst.

3. For LF-tags or catalog resources, choose Resources matched by LF-Tags (recommended).

4. Choose key Division and value analyst.

5. For Database permissions, select Describe under Database permissions.

6. For Table and column permissions, select Select and Describe under Table permissions.

7. Choose Grant.

8. Repeat these steps for user DataAnalyst, where the LF-Tag key is Confidentiality and value is public.

   At this point, the data analyst user in the consumer account should be able to find the database and resource link, and query the shared table via the Athena console at https://console.amazonaws.cn/athena/. If not, confirm if the following are properly configured:

   • The resource link is created for the shared table

   • You granted the user access to the LF-Tag shared by the producer account

   • You granted the user access to the LF-Tag associated to the resource link and database that the resource link is created in

   • Check if you assigned the correct LF-Tag to the resource link, and to the database that the resource link is created in

Revoke permission for IAMAllowedPrincipals on the database, tables, and columns (Optional)

• At the very beginning of this tutorial, we changed Lake Formation Data Catalog settings. If you skipped that part, this step is required. For instructions, see the optional step in the previous section.

Grant data permission to the consumer account

1. Note

   If you’re signed in to producer account as another user, sign out first.

   Sign into the Lake Formation console at https://console.amazonaws.cn/lakeformation/ using the producer account data lake administrator using the Amazon Web Services account ID, IAM user name (default is DatalakeAdminProducer), and password specified during Amazon CloudFormation stack creation.

2. On the Permissions page, under Data lake Permissions choose Grant.

3. Under Principals, choose External accounts, and enter one or more Amazon Web Services account IDs or Amazon organizations IDs. For more information see: Amazon Organizations.

   Organizations that the producer account belongs to and Amazon Web Services accounts within the same organization appear automatically. Otherwise, manually enter the account ID or organization ID.

4. For LF-Tags or catalog resources, choose Named data catalog resources.

5. Under Databases, choose the database lakeformation_tutorial_cross_account_database_named_resource.

6. Choose Add LF-Tag.

7. Under Tables, choose All tables.

8. For Table column permissions¸ choose Select, and Describe under Table permissions.

9. SelectSelect and Describe, under Grantable Permissions.

10. Optionally, for Data permissions, choose Simple column-based access if column-level permission management is required.

11. Choose Grant.

Accept a resource share from Amazon RAM

1. Sign into the Amazon console at https://console.amazonaws.cn/connect/ using the consumer account data lake administrator using the IAM user name (default is DatalakeAdminConsumer) and password specified during Amazon CloudFormation stack creation.

2. On the Amazon RAM console, in the navigation pane, under Shared with me, Resource shares, choose the shared Lake Formation resource. The Status should be Pending.

3. Choose Action and Grant.

4. Confirm the resource details, and choose Accept resource share.

   At this point, the consumer account data lake administrator should be able to find the shared resource on the Lake Formation console (https://console.amazonaws.cn/lakeformation/) under Data Catalog, Databases.

Create a resource link for the shared table

• Follow the instructions in Step 3: Implement cross-account sharing using the tag-based access control method (step 6) to create a resource link for a shared table. Name the resource link amazon_reviews_table_named_resource_resource_link. Create the resource link in the database lakeformation_tutorial_cross_account_database_consumer.

Grant data permission for the shared table to the consumer

To grant data permission for the shared table to the consumer, complete the following steps:

1. On the Lake Formationconsole (https://console.amazonaws.cn/lakeformation/), under Permissions, Data lake permissions, choose Grant.

2. For Principals, choose IAM users and roles, and choose the user DataAnalyst.

3. For LF-Tags or catalog resources, choose Named data catalog resources.

4. Under Databases, choose the database lakeformation_tutorial_cross_account_database_named_resource. If you don’t see the database on the drop-down list, choose Load more.

5. Under Tables, choose the table amazon_reviews_table_named_resource.

6. For Table and column permissions, select Select and Describe under Table permissions.

7. Choose Grant.

Grant data permission for the resource link to the consumer

In addition to granting the data lake user permission to access the shared table, you also need to grant the data lake user permission to access the resource link.

1. On the Lake Formation console (https://console.amazonaws.cn/lakeformation/), under Permissions, Data lake permissions, choose Grant.

2. For Principals, choose IAM users and roles, and choose the user DataAnalyst.

3. For LF-Tags or catalog resources, choose Named data catalog resources.

4. Under Databases, choose the database lakeformation_tutorial_cross_account_database_consumer. If you don’t see the database on the drop-down list, choose Load more.

5. Under Tables, choose the table amazon_reviews_table_named_resource_resource_link.

6. For Resource link permissions, select Describe under Resource link permissions.

7. Choose Grant.

   At this point, the data analyst user in the consumer account should be able to find the database and resource link, and query the shared table via the Athena console.

   If not, confirm if the following are properly configured:

   • The resource link is created for the shared table

   • You granted the user access to the table shared by the producer account

   • You granted the user access to the resource link and database for which the resource link is created

1. Sign into Lake Formation console at https://console.amazonaws.cn/lakeformation/ using the producer account and delete or change the following:

   • Amazon Resource Access Manager resource share

   • Lake Formation tags

   • Amazon CloudFormation stack

   • Lake Formation settings

   • Amazon Glue Data Catalog

2. Sign into Lake Formation console at https://console.amazonaws.cn/lakeformation/ using the consumer account and delete or change the following:

   • Lake Formation tags

   • Amazon CloudFormation stack