Document history for Amazon Lake Formation
The following table describes important changes to the documentation for Amazon Lake Formation.
| Change | Description | Date |
|---|---|---|
Updated policy change | Documented the change (added statement IDs and removed redundant permissions) to the AWSLakeFormationCrossAccountManager and AWSLakeFormationDataAdmin policies. | March 14, 2024 |
Updated setting up Lake Formation | Updated the steps in the Set up Amazon Lake Formation section. | February 7, 2024 |
Updated policy change | Added new permissions to the service-linked role's inline policy. For more information, see Using service-linked roles for Lake Formation. | February 7, 2024 |
Updated policy change | Documented the change to the LakeFormationDataAccessServiceRolePolicy policy. | February 2, 2024 |
Consolidated Lake Formation limitations | Created a unified section for Lake Formation limitations and considerations. For more information, see Lake Formation limitations. | December 15, 2023 |
Added documentation for Iceberg compaction | For better read performance by Amazon analytics services such as Athena and Amazon EMR, and Amazon Glue ETL jobs, Amazon Glue Data Catalog provides managed compaction (a process that compacts small Amazon S3 objects into larger objects) for Iceberg tables in the Data Catalog. For more information, see Optimizing Iceberg tables. | November 25, 2023 |
Added documentation for IAM Identity Center integration | IAM Identity Center integrations allows users and groups to access Data Catalog resources enforcing Lake Formation permissions. For more information, see IAM Identity Center integration. | November 25, 2023 |
Added documentation for Data Catalog views | You can create views in the Amazon Glue Data Catalog that references up to 10 tables using SQL editors for Amazon Athena or Amazon Redshift. For more information, see Creating views. | November 25, 2023 |
Updated the policy change | Documented the change to the AWSLakeFormationCrossAccountManager policy. | October 25, 2023 |
Added documentation for hybrid access mode | Hybrid access mode provides the flexibility to selectively enable Lake Formation permissions for databases and tables in your Amazon Glue Data Catalog. With hybrid access mode, you now have an incremental path that allows you to set Lake Formation permissions for a specific set of users without interrupting the permission policies of other existing users or workloads. For more information, see Hybrid access mode. | September 26, 2023 |
Added documentation for creating Apache Iceberg tables | You can now create Apache Iceberg tables that use the Apache Parquet data format in the Amazon Glue Data Catalog with data residing in Amazon S3. For more information, see Creating Iceberg tables. | August 16, 2023 |
Added documentation for cross-Region data access | Lake Formation supports querying Data Catalog tables across Amazon Regions. You can access data in a Region from other Regions using Athena, Amazon EMR, and run Amazon Glue ETL by creating resource links in other Regions pointing to the source databases and tables. You can connect the Data Catalog to external metastores that store metadata for your Amazon S3 data, and securely manage data access permissions using Amazon Lake Formation. For more information, see Accessing tables across Regions. | June 30, 2023 |
Re-organized contents | Re-organized chapters in the guide to match Lake Formation user journey. | May 15, 2023 |
Added documentation for HMS federation | You can connect the Data Catalog to external metastores that store metadata for your Amazon S3 data, and securely manage data access permissions using Amazon Lake Formation. For more information, see Managing permissions on datasets that use external metastores. | April 15, 2023 |
Added documentation for Amazon Redshift data sharing | You can now securely manage data in a datashare from Amazon Redshift using Lake Formation permissions. Lake Formation supports licensing access to your data through Amazon Web Services Data Exchange. For more information, see Data sharing in Amazon Lake Formation. | November 30, 2022 |
Support for cross-account data sharing directly with principals | Added information about sharing data directly with IAM principals in another account. For more information see Cross-account data sharing in Amazon Lake Formation. | November 10, 2022 |
Support for Amazon RAM enabled data sharing using TBAC | Added information about The LF-TBAC method of granting Data Catalog permissions use Amazon Resource Access Manager for cross-account grants. | November 10, 2022 |
Added a section on working with other services | Added information on how Amazon services such as Athena, Amazon Glue, Redshift Spectrum, and Amazon EMR can use Lake Formation to securely access data in Amazon S3 locations registered with Lake Formation. For more information see Working with other Amazon services. | November 10, 2022 |
Document history for Amazon Lake Formation | Added information on troubleshooting an error when using Amazon EMR to access cross-account data. For more information, see Error when using Amazon EMR to access data shared via cross-account. | November 7, 2022 |
Updates to cross-account resource share | Added a description for how cross-account resource shares work in Lake Formation. Documented the change to the AWSLakeFormationCrossAccountManager policy. | May 6, 2022 |
New tutorials | Added new tutorials for creating governed tables, securing data lakes, and sharing data lakes. For more details, see Get started section. | April 20, 2022 |
New Lake Formation landing page | Updated the Lake Formation landing page to include links for tutorials that provide step-by-step instructions on how to build a data lake, ingest data, share, and secure data lakes using Lake Formation. | April 20, 2022 |
Support for credential vending | Added information about credential vending, which supports Lake Formation to allow third-party services to integrate with Lake Formation by using credential vending API operations. For more information, see How credential vending works in Lake Formation. | February 28, 2022 |
Support for governed tables and advanced data filtering | Added information about governed tables, which support ACID transactions, automatic data compaction, and time-travel queries. Added information about creating data filters to support for column-level security, row-level security, and cell-level security. For more information, see Governed Tables in Lake Formation and Data Filtering and Cell-Level Security in Lake Formation. | November 30, 2021 |
Support for VPC interface endpoints | Added information about creating a virtual private cloud (VPC) interface endpoint for Lake Formation, so that communication between your VPC and Lake Formation is conducted entirely and securely within the Amazon network. For more information, see Using Lake Formation with VPC Endpoints . | October 11, 2021 |
Support for VPC endpoint policies | Added information about support for Virtual Private Cloud (VPC) endpoint policies in Lake Formation. For more information, see Using Lake Formation with VPC Endpoints. | October 11, 2021 |
Support for tag-based access control | Lake Formation tag-based access control provides a new, more scalable way to manage access to Data Catalog resources and underlying data by using LF-Tags. For more information, see Lake Formation Tag-Based Access Control. | May 7, 2021 |
New opt-in requirement for data filtering on Amazon EMR. | Added information about the requirement to opt in to allow Amazon EMR to filter data that is managed by Lake Formation. For more information, see Allow Data Filtering on Amazon EMR. | October 9, 2020 |
Support for granting full cross-account permissions on Data Catalog databases | Added information about granting full Lake Formation permissions on Data Catalog databases across Amazon
accounts, including | October 1, 2020 |
Support for Amazon Athena users authenticating through SAML. | Added information about support for Athena users who connect through the JDBC or ODBC driver and authenticate through SAML identity providers such as Okta and Microsoft Active Directory Federation Service (AD FS). For more information, see Amazon Service Integrations with Lake Formation. | September 30, 2020 |
Support for cross-account access with an encrypted Data Catalog | Added information about granting cross-account permissions when the Data Catalog is encrypted. For more information, see Cross-Account Access Prerequisites. | July 30, 2020 |
Support for cross-account access to the data lake | Added information about granting Amazon Lake Formation permissions on Data Catalog databases and tables to external Amazon accounts and organizations, and about accessing Data Catalog objects shared from external accounts. For more information, see Cross-Account Access. | July 7, 2020 |
Integration with Amazon QuickSight | Added information about how to grant Lake Formation permissions to Amazon QuickSight Enterprise Edition users so that they may access datasets residing in registered Amazon S3 locations. For more information, see Granting Data Catalog Permissions. | June 29, 2020 |
Updates to setting up and Getting Started chapters | Reorganized and improved the Setting Up and Getting Started chapters. Updated the recommended Amazon Identity and Access Management (IAM) permissions for the data lake administrator. | February 27, 2020 |
Support for Amazon Key Management Service | Added information about how Lake Formation support for Amazon Key Management Service (Amazon KMS) simplifies setting up integrated services to read and write encrypted data in registered Amazon Simple Storage Service (Amazon S3) locations. Added information about how to register Amazon S3 locations that are encrypted with Amazon KMS keys. For more information, see Adding an Amazon S3 location to your data lake. | February 27, 2020 |
Updates to blueprints and data lake administrator IAM policies | Clarified input parameters for incremental database blueprints. Updated the IAM policies required for a data lake administrator. | December 20, 2019 |
Security chapter rewrite and upgrade chapter revisions | Improved the security and upgrading chapters. | October 29, 2019 |
Super permission replaces All permission | Updated the Security and Upgrading chapters to reflect the replacement of the permission
| October 10, 2019 |
Additions, corrections, and clarifications | Made additions, corrections, and clarifications based on feedback. Revised the security
chapter. Updated the Security and Upgrading chapters to reflect the replacement of the group
| September 11, 2019 |
New guide | This is the initial release of the Amazon Lake Formation Developer Guide. | August 8, 2019 |