Lake Formation examples using Amazon CLI
The following code examples show you how to perform actions and implement common scenarios by using the Amazon Command Line Interface with Lake Formation.
Actions are code excerpts from larger programs and must be run in context. While actions show you how to call individual service functions, you can see actions in context in their related scenarios and cross-service examples.
Scenarios are code examples that show you how to accomplish a specific task by calling multiple functions within the same service.
Each example includes a link to GitHub, where you can find instructions on how to set up and run the code in context.
Topics
Actions
The following code example shows how to use add-lf-tags-to-resource
.
- Amazon CLI
-
To attach one or more LF-tags to an existing resource
The following
add-lf-tags-to-resource
example attaches given LF-tag to the table resource.aws lakeformation add-lf-tags-to-resource \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "analyst" ] }] }
Output:
{ "Failures": [] }
For more information, see Assigning LF-Tags to Data Catalog resources
in the Amazon Lake Formation Developer Guide. -
For API details, see AddLfTagsToResource
in Amazon CLI Command Reference.
-
The following code example shows how to use batch-grant-permissions
.
- Amazon CLI
-
To bulk grant permissions on resources to the principals
The following
batch-grant-permissions
example bulk grants access on specified resources to the principals.aws lakeformation batch-grant-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Entries": [{ "Id": "1", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [ "ALL" ] }, { "Id": "2", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_customer" } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [ "ALL" ] }, { "Id": "3", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-business-analyst" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [ "ALL" ] }, { "Id": "4", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "DataCellsFilter": { "TableCatalogId": "123456789111", "DatabaseName": "tpc", "TableName": "dl_tpc_item", "Name": "developer_item" } }, "Permissions": [ "SELECT" ], "PermissionsWithGrantOption": [] } ] }
Output:
{ "Failures": [] }
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. -
For API details, see BatchGrantPermissions
in Amazon CLI Command Reference.
-
The following code example shows how to use batch-revoke-permissions
.
- Amazon CLI
-
To bulk revoke permissions on resources from the principals
The following
batch-revoke-permissions
example bulk revokes access on specified resources from the principals.aws lakeformation batch-revoke-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Entries": [{ "Id": "1", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [ "ALL" ] }, { "Id": "2", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-business-analyst" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [ "ALL" ] } ] }
Output:
{ "Failures": [] }
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. -
For API details, see BatchRevokePermissions
in Amazon CLI Command Reference.
-
The following code example shows how to use cancel-transaction
.
- Amazon CLI
-
To cancel a transaction
The following
cancel-transaction
example cancels the transaction.aws lakeformation cancel-transaction \ --transaction-id='b014d972ca8347b89825e33c5774aec4'
This command produces no output.
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see CancelTransaction
in Amazon CLI Command Reference.
-
The following code example shows how to use commit-transaction
.
- Amazon CLI
-
To commit transaction
The following
commit-transaction
example commits the transaction.aws lakeformation commit-transaction \ --transaction-id='b014d972ca8347b89825e33c5774aec4'
Output:
{ "TransactionStatus": "committed" }
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see CommitTransaction
in Amazon CLI Command Reference.
-
The following code example shows how to use create-data-cells-filter
.
- Amazon CLI
-
Example 1: To create data cell filter
The following
create-data-cells-filter
example creates a data cell filter to allow one to grant access to certain columns based on row condition.aws lakeformation create-data-cells-filter \ --cli-input-json
file://input.json
Contents of
input.json
:{ "TableData": { "ColumnNames": ["p_channel_details", "p_start_date_sk", "p_promo_name"], "DatabaseName": "tpc", "Name": "developer_promotion", "RowFilter": { "FilterExpression": "p_promo_name='ese'" }, "TableCatalogId": "123456789111", "TableName": "dl_tpc_promotion" } }
This command produces no output.
For more information, see Data filtering and cell-level security in Lake Formation
in the Amazon Lake Formation Developer Guide. Example 2: To create column filter
The following
create-data-cells-filter
example creates a data filter to allow one to grant access to certain columns.aws lakeformation create-data-cells-filter \ --cli-input-json
file://input.json
Contents of
input.json
:{ "TableData": { "ColumnNames": ["p_channel_details", "p_start_date_sk", "p_promo_name"], "DatabaseName": "tpc", "Name": "developer_promotion_allrows", "RowFilter": { "AllRowsWildcard": {} }, "TableCatalogId": "123456789111", "TableName": "dl_tpc_promotion" } }
This command produces no output.
For more information, see Data filtering and cell-level security in Lake Formation
in the Amazon Lake Formation Developer Guide. Example 3: To create data filter with exclude columns
The following
create-data-cells-filter
example creates a data filter to allow one to grant access all except the mentioned columns.aws lakeformation create-data-cells-filter \ --cli-input-json
file://input.json
Contents of
input.json
:{ "TableData": { "ColumnWildcard": { "ExcludedColumnNames": ["p_channel_details", "p_start_date_sk"] }, "DatabaseName": "tpc", "Name": "developer_promotion_excludecolumn", "RowFilter": { "AllRowsWildcard": {} }, "TableCatalogId": "123456789111", "TableName": "dl_tpc_promotion" } }
This command produces no output.
For more information, see Data filtering and cell-level security in Lake Formation
in the Amazon Lake Formation Developer Guide. -
For API details, see CreateDataCellsFilter
in Amazon CLI Command Reference.
-
The following code example shows how to use create-lf-tag
.
- Amazon CLI
-
To create LF-Tag
The following
create-lf-tag
example creates an LF-Tag with the specified name and values.aws lakeformation create-lf-tag \ --catalog-id '
123456789111
' \ --tag-key 'usergroup
' \ --tag-values '["developer","analyst","campaign"]
'This command produces no output.
For more information, see Managing LF-Tags for metadata access control
in the Amazon Lake Formation Developer Guide. -
For API details, see CreateLfTag
in Amazon CLI Command Reference.
-
The following code example shows how to use delete-data-cells-filter
.
- Amazon CLI
-
To delete data cell filter
The following
delete-data-cells-filter
example deletes given data cell filter.aws lakeformation delete-data-cells-filter \ --cli-input-json
file://input.json
Contents of
input.json
:{ "TableCatalogId": "123456789111", "DatabaseName": "tpc", "TableName": "dl_tpc_promotion", "Name": "developer_promotion" }
This command produces no output.
For more information, see Data filtering and cell-level security in Lake Formation
in the Amazon Lake Formation Developer Guide. -
For API details, see DeleteDataCellsFilter
in Amazon CLI Command Reference.
-
The following code example shows how to use delete-lf-tag
.
- Amazon CLI
-
To delete LF-Tag definition
The following
delete-lf-tag
example deletes LF-Tag definition.aws lakeformation delete-lf-tag \ --catalog-id '
123456789111
' \ --tag-key 'usergroup
'This command produces no output.
For more information, see Managing LF-Tags for metadata access control
in the Amazon Lake Formation Developer Guide. -
For API details, see DeleteLfTag
in Amazon CLI Command Reference.
-
The following code example shows how to use delete-objects-on-cancel
.
- Amazon CLI
-
To delete object when transaction is cancelled
The following
delete-objects-on-cancel
example deletes the listed s3 object when the transaction is cancelled.aws lakeformation delete-objects-on-cancel \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "012345678901", "DatabaseName": "tpc", "TableName": "dl_tpc_household_demographics_gov", "TransactionId": "1234d972ca8347b89825e33c5774aec4", "Objects": [{ "Uri": "s3://lf-data-lake-012345678901/target/dl_tpc_household_demographics_gov/run-unnamed-1-part-block-0-r-00000-snappy-ff26b17504414fe88b302cd795eabd00.parquet", "ETag": "1234ab1fc50a316b149b4e1f21a73800" }] }
This command produces no output.
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see DeleteObjectsOnCancel
in Amazon CLI Command Reference.
-
The following code example shows how to use deregister-resource
.
- Amazon CLI
-
To deregister data lake storage
The following
deregister-resource
example deregisters the resource as managed by the Lake Formation.aws lakeformation deregister-resource \ --cli-input-json
file://input.json
Contents of
input.json
:{ "ResourceArn": "arn:aws:s3:::lf-emr-athena-result-123" }
This command produces no output.
For more information, see Adding an Amazon S3 location to your data lake
in the Amazon Lake Formation Developer Guide. -
For API details, see DeregisterResource
in Amazon CLI Command Reference.
-
The following code example shows how to use describe-transaction
.
- Amazon CLI
-
To retrieve a transaction details
The following
describe-transaction
example returns the details of a single transaction.aws lakeformation describe-transaction \ --transaction-id='8cb4b1a7cc8d486fbaca9a64e7d9f5ce'
Output:
{ "TransactionDescription": { "TransactionId": "12345972ca8347b89825e33c5774aec4", "TransactionStatus": "committed", "TransactionStartTime": "2022-08-10T14:29:04.046000+00:00", "TransactionEndTime": "2022-08-10T14:29:09.681000+00:00" } }
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see DescribeTransaction
in Amazon CLI Command Reference.
-
The following code example shows how to use extend-transaction
.
- Amazon CLI
-
To extend a transaction
The following
extend-transaction
example extends the transaction.aws lakeformation extend-transaction \ --transaction-id='8cb4b1a7cc8d486fbaca9a64e7d9f5ce'
This command produces no output.
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see ExtendTransaction
in Amazon CLI Command Reference.
-
The following code example shows how to use get-data-lake-settings
.
- Amazon CLI
-
To retrieve Amazon Lake Formation-managed data lake settings
The following
get-data-lake-settings
example retrieves the list of data lake administrators and other data lake settings.aws lakeformation get-data-lake-settings \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111" }
Output:
{ "DataLakeSettings": { "DataLakeAdmins": [{ "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-admin" }], "CreateDatabaseDefaultPermissions": [], "CreateTableDefaultPermissions": [ { "Principal": { "DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS" }, "Permissions": [ "ALL" ] } ], "TrustedResourceOwners": [], "AllowExternalDataFiltering": true, "ExternalDataFilteringAllowList": [{ "DataLakePrincipalIdentifier": "123456789111" }], "AuthorizedSessionTagValueList": [ "Amazon EMR" ] } }
For more information, see Changing the default security settings for your data lake
in the Amazon Lake Formation Developer Guide. -
For API details, see GetDataLakeSettings
in Amazon CLI Command Reference.
-
The following code example shows how to use get-effective-permissions-for-path
.
- Amazon CLI
-
To retrieve permissions on resources located at specific path
The following
get-effective-permissions-for-path
example returns the Lake Formation permissions for a specified table or database resource located at a path in Amazon S3.aws lakeformation get-effective-permissions-for-path \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "ResourceArn": "arn:aws:s3:::lf-data-lake-123456789111" }
Output:
{ "Permissions": [{ "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-campaign-manager" }, "Resource": { "Database": { "Name": "tpc" } }, "Permissions": [ "DESCRIBE" ], "PermissionsWithGrantOption": [] }, { "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:role/EMR-RuntimeRole" }, "Resource": { "Database": { "Name": "tpc" } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [] }, { "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:saml-provider/oktaSAMLProvider:user/emr-developer" }, "Resource": { "Database": { "Name": "tpc" } }, "Permissions": [ "ALL", "DESCRIBE" ], "PermissionsWithGrantOption": [] }, { "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-admin" }, "Resource": { "Database": { "Name": "tpc" } }, "Permissions": [ "ALL", "ALTER", "CREATE_TABLE", "DESCRIBE", "DROP" ], "PermissionsWithGrantOption": [ "ALL", "ALTER", "CREATE_TABLE", "DESCRIBE", "DROP" ] }, { "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:role/LF-GlueServiceRole" }, "Resource": { "Database": { "Name": "tpc" } }, "Permissions": [ "CREATE_TABLE" ], "PermissionsWithGrantOption": [] } ], "NextToken": "E5SlJDSTZleUp6SWpvaU9UQTNORE0zTXpFeE5Ua3pJbjE5TENKbGVIQnBjbUYwYVc5dUlqcDdJbk5sWTI5dVpITWlPakUyTm==" }
For more information, see Managing Lake Formation permissions
in the Amazon Lake Formation Developer Guide. -
For API details, see GetEffectivePermissionsForPath
in Amazon CLI Command Reference.
-
The following code example shows how to use get-lf-tag
.
- Amazon CLI
-
To retrieve LF-tag definition
The following
get-lf-tag
example retrieves LF-tag definition.aws lakeformation get-lf-tag \ --catalog-id '
123456789111
' \ --tag-key 'usergroup
'Output:
{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "analyst", "campaign", "developer" ] }
For more information, see Managing LF-Tags for metadata access control
in the Amazon Lake Formation Developer Guide. -
For API details, see GetLfTag
in Amazon CLI Command Reference.
-
The following code example shows how to use get-query-state
.
- Amazon CLI
-
To retrieve state of a submitted query
The following
get-query-state
example returns the state of a query previously submitted.aws lakeformation get-query-state \ --query-id='1234273f-4a62-4cda-8d98-69615ee8be9b'
Output:
{ "State": "FINISHED" }
For more information, see Transactional data operations
in the Amazon Lake Formation Developer Guide. -
For API details, see GetQueryState
in Amazon CLI Command Reference.
-
The following code example shows how to use get-query-statistics
.
- Amazon CLI
-
To retrieve query statistics
The following
get-query-statistics
example retrieves statistics on the planning and execution of a query.aws lakeformation get-query-statistics \ --query-id='1234273f-4a62-4cda-8d98-69615ee8be9b'
Output:
{ "ExecutionStatistics": { "AverageExecutionTimeMillis": 0, "DataScannedBytes": 0, "WorkUnitsExecutedCount": 0 }, "PlanningStatistics": { "EstimatedDataToScanBytes": 43235, "PlanningTimeMillis": 2377, "QueueTimeMillis": 440, "WorkUnitsGeneratedCount": 1 }, "QuerySubmissionTime": "2022-08-11T02:14:38.641870+00:00" }
For more information, see Transactional data operations
in the Amazon Lake Formation Developer Guide. -
For API details, see GetQueryStatistics
in Amazon CLI Command Reference.
-
The following code example shows how to use get-resource-lf-tags
.
- Amazon CLI
-
To list LF-tags
The following
list-lf-tags
example returns list of LF-tags that the requester has permission to view.aws lakeformation list-lf-tags \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "ResourceShareType": "ALL", "MaxResults": 2 }
Output:
{ "LFTags": [{ "CatalogId": "123456789111", "TagKey": "category", "TagValues": [ "private", "public" ] }, { "CatalogId": "123456789111", "TagKey": "group", "TagValues": [ "analyst", "campaign", "developer" ] }], "NextToken": "kIiwiZXhwaXJhdGlvbiI6eyJzZWNvbmRzIjoxNjYwMDY4dCI6ZmFsc2V9" }
For more information, see Managing LF-Tags for metadata access control
in the Amazon Lake Formation Developer Guide. -
For API details, see GetResourceLfTags
in Amazon CLI Command Reference.
-
The following code example shows how to use get-table-objects
.
- Amazon CLI
-
To list objects of governed table
The following
get-table-objects
example returns the set of Amazon S3 objects that make up the specified governed table.aws lakeformation get-table-objects \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "012345678901", "DatabaseName": "tpc", "TableName": "dl_tpc_household_demographics_gov", "QueryAsOfTime": "2022-08-10T15:00:00" }
Output:
{ "Objects": [{ "PartitionValues": [], "Objects": [{ "Uri": "s3://lf-data-lake-012345678901/target/dl_tpc_household_demographics_gov/run-unnamed-1-part-block-0-r-00000-snappy-ff26b17504414fe88b302cd795eabd00.parquet", "ETag": "12345b1fc50a316b149b4e1f21a73800", "Size": 43235 }] }] }
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see GetTableObjects
in Amazon CLI Command Reference.
-
The following code example shows how to use get-work-unit-results
.
- Amazon CLI
-
To retrieve work units of given query
The following
get-work-unit-results
example returns the work units resulting from the query.aws lakeformation get-work-units \ --query-id='1234273f-4a62-4cda-8d98-69615ee8be9b' \ --work-unit-id '
0
' \ --work-unit-token 'B2fMSdmQXe9umX8Ux8XCo4=
'outfile
Output:
outfile with Blob content.
For more information, see Transactional data operations
in the Amazon Lake Formation Developer Guide. -
For API details, see GetWorkUnitResults
in Amazon CLI Command Reference.
-
The following code example shows how to use get-work-units
.
- Amazon CLI
-
To retrieve work units
The following
get-work-units
example retrieves the work units generated by the StartQueryPlanning operation.aws lakeformation get-work-units \ --query-id='1234273f-4a62-4cda-8d98-69615ee8be9b'
Output:
{ "WorkUnitRanges": [{ "WorkUnitIdMax": 0, "WorkUnitIdMin": 0, "WorkUnitToken": "1234eMAk4kLO4umqEL4Z5WuxL04AXwABABVhd3MtY3J5cHRvLXB1YmxpYy1rZXkAREEwYm9QbkhINmFYTWphbmMxZW1PQmEyMGlUb0JFbXNlWmRYc0NmckRIR1dmQ0hjY2YzNFdMcmNXb2JGZmhEK0QvZz09AAEAB2F3cy1rbXMAS2Fybjphd3M6a21zOnVzLWVhc3QtMTo3MDkxNTAyNDkyNDk6a2V5L2VmYmI3NDUyLTY1MjYtNGJiOS1iNmZhLTEzYzJkMTM3MmU2OQC4AQIBAHg6eWNF2ZrQATTAuPDJVCEAQSyIF67vX+f88jzGrYq22gE6jkQlpOB+Oet2eqNUmFudAAAAfjB8BgkqhkiG9w0BBwagbzBtAgEAMGgGCSqGSIb3DQEHATAeBglghkgBZQMEAS4wEQQMCOEWRdafowek3RUmAgEQgDsYZZE84nnnbNmvsqCBPLh19nLQ10mUWOg9IfiaOwefEn6L920V0x1LpJACo7MtIBLXnbGcz2dFDZjFygIAAAAADAAAEAAAAAAAAAAAAAAAAAAQSQf8XDSI5pvR4Fx4JsrS/////wAAAAEAAAAAAAAAAAAAAAEAAACX3/w5h75QAPomfKH+cyEKYU1yccUmBl+VSojiGOtdsUk7vcjYXUUboYm3dvqRqX2s4gROMOn+Ij8R0/8jYmnHkpvyAFNVRPyETyIKg7k5Z9+5I1c2d3446Jw/moWGGxjH8AEG9h27ytmOhozxDOEi/F2ZoXz6wlGDfGUo/2WxCkYOhTyNaw6TM+7drTM7yrW4iNVLUM0LX0xnFjIAhLhooWJek6vjQZUAZzBlAjBH8okRtYP8R7AY2Wls/hqFBhG0V4l42AC0LxsuZbMQrE2SzWZUZ0E9Uew7/n0cyX4CMQDR79INyv4ysMByW9kKGGKyba+cCNklExMR+btBQBmMuB2fMSdmQXe9umX8Ux8XCo4=" }], "QueryId": "1234273f-4a62-4cda-8d98-69615ee8be9b" }
For more information, see Transactional data operations
in the Amazon Lake Formation Developer Guide. -
For API details, see GetWorkUnits
in Amazon CLI Command Reference.
-
The following code example shows how to use grant-permissions
.
- Amazon CLI
-
Example 1: To grant permissions to the principal on resources using LF-Tags
The following
grant-permissions
example grants ALL permissions to the principal on database resource that matches the LF-Tag policy.aws lakeformation grant-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-admin" }, "Resource": { "LFTagPolicy": { "CatalogId": "123456789111", "ResourceType": "DATABASE", "Expression": [{ "TagKey": "usergroup", "TagValues": [ "analyst", "developer" ] }] } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [ "ALL" ] }
This command produces no output.
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. Example 2: To grant column level permissions to the principal
The following
grant-permissions
example grants permission to select specific column to the principal.aws lakeformation grant-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "TableWithColumns": { "CatalogId": "123456789111", "ColumnNames": ["p_end_date_sk"], "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "Permissions": [ "SELECT" ], "PermissionsWithGrantOption": [] }
This command produces no output.
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. Example 3: To grant table permissions to the principal
The following
grant-permissions
example grants select permission on all tables of given database to the principal.aws lakeformation grant-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "TableWildcard": {} } }, "Permissions": [ "SELECT" ], "PermissionsWithGrantOption": [] }
This command produces no output.
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. Example 4: To grant permissions on LF-Tags to the principal
The following
grant-permissions
example grants associate permission on LF-Tags to the principal.aws lakeformation grant-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "LFTag": { "CatalogId": "123456789111", "TagKey": "category", "TagValues": [ "private", "public" ] } }, "Permissions": [ "ASSOCIATE" ], "PermissionsWithGrantOption": [] }
This command produces no output.
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. Example 5: To grant permissions on data locations to the principal
The following
grant-permissions
example grants permission on data location to the principal.aws lakeformation grant-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "DataLocation": { "CatalogId": "123456789111", "ResourceArn": "arn:aws:s3:::lf-data-lake-123456789111" } }, "Permissions": [ "DATA_LOCATION_ACCESS" ], "PermissionsWithGrantOption": [] }
This command produces no output.
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. -
For API details, see GrantPermissions
in Amazon CLI Command Reference.
-
The following code example shows how to use list-data-cells-filter
.
- Amazon CLI
-
To list data cell filters
The following
list-data-cells-filter
example list data cell filter for given table.aws lakeformation list-data-cells-filter \ --cli-input-json
file://input.json
Contents of
input.json
:{ "MaxResults": 2, "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }
Output:
{ "DataCellsFilters": [{ "TableCatalogId": "123456789111", "DatabaseName": "tpc", "TableName": "dl_tpc_promotion", "Name": "developer_promotion", "RowFilter": { "FilterExpression": "p_promo_name='ese'" }, "ColumnNames": [ "p_channel_details", "p_start_date_sk", "p_purpose", "p_promo_id", "p_promo_name", "p_end_date_sk", "p_discount_active" ] }, { "TableCatalogId": "123456789111", "DatabaseName": "tpc", "TableName": "dl_tpc_promotion", "Name": "developer_promotion_allrows", "RowFilter": { "FilterExpression": "TRUE", "AllRowsWildcard": {} }, "ColumnNames": [ "p_channel_details", "p_start_date_sk", "p_promo_name" ] } ], "NextToken": "2MDA2MTgwNiwibmFub3MiOjE0MDAwMDAwMH19" }
For more information, see Data filtering and cell-level security in Lake Formation
in the Amazon Lake Formation Developer Guide. -
For API details, see ListDataCellsFilter
in Amazon CLI Command Reference.
-
The following code example shows how to use list-permissions
.
- Amazon CLI
-
Example 1: To retrieve list of principal permissions on the resource
The following
list-permissions
example returns a list of principal permissions on the database resources.aws lakeformation list-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "ResourceType": "DATABASE", "MaxResults": 2 }
Output:
{ "PrincipalResourcePermissions": [{ "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-campaign-manager" }, "Resource": { "Database": { "CatalogId": "123456789111", "Name": "tpc" } }, "Permissions": [ "DESCRIBE" ], "PermissionsWithGrantOption": [] }], "NextToken": "E5SlJDSTZleUp6SWpvaU9UQTNORE0zTXpFeE5Ua3pJbjE5TENKbGVIQnBjbUYwYVc5dUlqcDdJbk5sWTI5dVpITWlPakUyTm" }
For more information, see Managing Lake Formation permissions
in the Amazon Lake Formation Developer Guide. Example 2: To retrieve list of principal permissions on the table with data filters
The following
list-permissions
example list the permissions on the table with related data filters granted to the principal.aws lakeformation list-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_customer" } }, "IncludeRelated": "TRUE", "MaxResults": 10 }
Output:
{ "PrincipalResourcePermissions": [{ "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:role/Admin" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "customer", "Name": "customer_invoice" } }, "Permissions": [ "ALL", "ALTER", "DELETE", "DESCRIBE", "DROP", "INSERT" ], "PermissionsWithGrantOption": [ "ALL", "ALTER", "DELETE", "DESCRIBE", "DROP", "INSERT" ] }, { "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:role/Admin" }, "Resource": { "TableWithColumns": { "CatalogId": "123456789111", "DatabaseName": "customer", "Name": "customer_invoice", "ColumnWildcard": {} } }, "Permissions": [ "SELECT" ], "PermissionsWithGrantOption": [ "SELECT" ] }, { "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:role/Admin" }, "Resource": { "DataCellsFilter": { "TableCatalogId": "123456789111", "DatabaseName": "customer", "TableName": "customer_invoice", "Name": "dl_us_customer" } }, "Permissions": [ "DESCRIBE", "SELECT", "DROP" ], "PermissionsWithGrantOption": [] } ], "NextToken": "VyeUFjY291bnRQZXJtaXNzaW9ucyI6ZmFsc2V9" }
For more information, see Managing Lake Formation permissions
in the Amazon Lake Formation Developer Guide. Example 3: To retrieve list of principal permissions on the LF-Tags
The following
list-permissions
example list the permissions on the LF-Tags granted to the principal.aws lakeformation list-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Resource": { "LFTag": { "CatalogId": "123456789111", "TagKey": "category", "TagValues": [ "private" ] } }, "MaxResults": 10 }
Output:
{ "PrincipalResourcePermissions": [{ "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-admin" }, "Resource": { "LFTag": { "CatalogId": "123456789111", "TagKey": "category", "TagValues": [ "*" ] } }, "Permissions": [ "DESCRIBE" ], "PermissionsWithGrantOption": [ "DESCRIBE" ] }, { "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-admin" }, "Resource": { "LFTag": { "CatalogId": "123456789111", "TagKey": "category", "TagValues": [ "*" ] } }, "Permissions": [ "ASSOCIATE" ], "PermissionsWithGrantOption": [ "ASSOCIATE" ] } ], "NextToken": "EJwY21GMGFXOXVJanA3SW5Ocm1pc3Npb25zIjpmYWxzZX0=" }
For more information, see Managing Lake Formation permissions
in the Amazon Lake Formation Developer Guide. -
For API details, see ListPermissions
in Amazon CLI Command Reference.
-
The following code example shows how to use list-resources
.
- Amazon CLI
-
To lists the resources managed by the Lake Formation
The following
list-resources
example lists the resources matching the condition that is managed by the Lake Formation.aws lakeformation list-resources \ --cli-input-json
file://input.json
Contents of
input.json
:{ "FilterConditionList": [{ "Field": "ROLE_ARN", "ComparisonOperator": "CONTAINS", "StringValueList": [ "123456789111" ] }], "MaxResults": 10 }
Output:
{ "ResourceInfoList": [{ "ResourceArn": "arn:aws:s3:::lf-data-lake-123456789111", "RoleArn": "arn:aws:iam::123456789111:role/LF-GlueServiceRole", "LastModified": "2022-07-21T02:12:46.669000+00:00" }, { "ResourceArn": "arn:aws:s3:::lf-emr-test-123456789111", "RoleArn": "arn:aws:iam::123456789111:role/EMRLFS3Role", "LastModified": "2022-07-29T16:22:03.211000+00:00" } ] }
For more information, see Managing Lake Formation permissions
in the Amazon Lake Formation Developer Guide. -
For API details, see ListResources
in Amazon CLI Command Reference.
-
The following code example shows how to use list-transactions
.
- Amazon CLI
-
To list all transactions details
The following
list-transactions
example returns metadata about transactions and their status.aws lakeformation list-transactions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "StatusFilter": "ALL", "MaxResults": 3 }
Output:
{ "Transactions": [{ "TransactionId": "1234569f08804cb790d950d4d0fe485e", "TransactionStatus": "committed", "TransactionStartTime": "2022-08-10T14:32:29.220000+00:00", "TransactionEndTime": "2022-08-10T14:32:33.751000+00:00" }, { "TransactionId": "12345972ca8347b89825e33c5774aec4", "TransactionStatus": "committed", "TransactionStartTime": "2022-08-10T14:29:04.046000+00:00", "TransactionEndTime": "2022-08-10T14:29:09.681000+00:00" }, { "TransactionId": "12345daf6cb047dbba8ad9b0414613b2", "TransactionStatus": "committed", "TransactionStartTime": "2022-08-10T13:56:51.261000+00:00", "TransactionEndTime": "2022-08-10T13:56:51.547000+00:00" } ], "NextToken": "77X1ebypsI7os+X2lhHsZLGNCDK3nNGpwRdFpicSOHgcX1/QMoniUAKcpR3kj3ts3PVdMA==" }
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see ListTransactions
in Amazon CLI Command Reference.
-
The following code example shows how to use put-data-lake-settings
.
- Amazon CLI
-
To set Amazon Lake Formation-managed data lake settings
The following
put-data-lake-settings
example sets the list of data lake administrators and other data lake settings.aws lakeformation put-data-lake-settings \ --cli-input-json
file://input.json
Contents of
input.json
:{ "DataLakeSettings": { "DataLakeAdmins": [{ "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-admin" } ], "CreateDatabaseDefaultPermissions": [], "CreateTableDefaultPermissions": [], "TrustedResourceOwners": [], "AllowExternalDataFiltering": true, "ExternalDataFilteringAllowList": [{ "DataLakePrincipalIdentifier ": "123456789111" }], "AuthorizedSessionTagValueList": ["Amazon EMR"] } }
This command produces no output.
For more information, see Changing the default security settings for your data lake
in the Amazon Lake Formation Developer Guide. -
For API details, see PutDataLakeSettings
in Amazon CLI Command Reference.
-
The following code example shows how to use register-resource
.
- Amazon CLI
-
Example 1: To register data lake storage using Service Linked Role
The following
register-resource
example registers the resource as managed by the Lake Formation using Service linked role.aws lakeformation register-resource \ --cli-input-json
file://input.json
Contents of
input.json
:{ "ResourceArn": "arn:aws:s3:::lf-emr-athena-result-123", "UseServiceLinkedRole": true }
This command produces no output.
For more information, see Adding an Amazon S3 location to your data lake
in the Amazon Lake Formation Developer Guide. Example 2: To register data lake storage using custom role
The following
register-resource
example registers the resource as managed by the Lake Formation using custom role.aws lakeformation register-resource \ --cli-input-json
file://input.json
Contents of
input.json
:{ "ResourceArn": "arn:aws:s3:::lf-emr-athena-result-123", "UseServiceLinkedRole": false, "RoleArn": "arn:aws:iam::123456789111:role/LF-GlueServiceRole" }
This command produces no output.
For more information, see Adding an Amazon S3 location to your data lake
in the Amazon Lake Formation Developer Guide. -
For API details, see RegisterResource
in Amazon CLI Command Reference.
-
The following code example shows how to use remove-lf-tags-from-resource
.
- Amazon CLI
-
To remove LF-Tag from a resource
The following
remove-lf-tags-from-resource
example removes the LF-Tag association with the table resource.aws lakeformation remove-lf-tags-from-resource \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }
Output:
{ "Failures": [] }
For more information, see Assigning LF-Tags to Data Catalog resources
in the Amazon Lake Formation Developer Guide. -
For API details, see RemoveLfTagsFromResource
in Amazon CLI Command Reference.
-
The following code example shows how to use revoke-permissions
.
- Amazon CLI
-
To revoke permissions on resources from the principal
The following
revoke-permissions
example revoke principal access to specific table of a given database.aws lakeformation revoke-permissions \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "123456789111", "Principal": { "DataLakePrincipalIdentifier": "arn:aws:iam::123456789111:user/lf-developer" }, "Resource": { "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_promotion" } }, "Permissions": [ "ALL" ], "PermissionsWithGrantOption": [] }
This command produces no output.
For more information, see Granting and revoking permissions on Data Catalog resources
in the Amazon Lake Formation Developer Guide. -
For API details, see RevokePermissions
in Amazon CLI Command Reference.
-
The following code example shows how to use search-databases-by-lf-tags
.
- Amazon CLI
-
To search on database resources by LFTags
The following
search-databases-by-lf-tags
example search on database resources matching LFTag expression.aws lakeformation search-databases-by-lf-tags \ --cli-input-json
file://input.json
Contents of
input.json
:{ "MaxResults": 1, "CatalogId": "123456789111", "Expression": [{ "TagKey": "usergroup", "TagValues": [ "developer" ] }] }
Output:
{ "DatabaseList": [{ "Database": { "CatalogId": "123456789111", "Name": "tpc" }, "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }] }
For more information, see Viewing the resources that a LF-Tag is assigned to
in the Amazon Lake Formation Developer Guide. -
For API details, see SearchDatabasesByLfTags
in Amazon CLI Command Reference.
-
The following code example shows how to use search-tables-by-lf-tags
.
- Amazon CLI
-
To search on table resources by LFTags
The following
search-tables-by-lf-tags
example search on table resources matching LFTag expression.aws lakeformation search-tables-by-lf-tags \ --cli-input-json
file://input.json
Contents of
input.json
:{ "MaxResults": 2, "CatalogId": "123456789111", "Expression": [{ "TagKey": "usergroup", "TagValues": [ "developer" ] }] }
Output:
{ "NextToken": "c2VhcmNoQWxsVGFnc0luVGFibGVzIjpmYWxzZX0=", "TableList": [{ "Table": { "CatalogId": "123456789111", "DatabaseName": "tpc", "Name": "dl_tpc_item" }, "LFTagOnDatabase": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }], "LFTagsOnTable": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }], "LFTagsOnColumns": [{ "Name": "i_item_desc", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_container", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_wholesale_cost", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_manufact_id", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_brand_id", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_formulation", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_current_price", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_size", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_rec_start_date", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_manufact", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_item_sk", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_manager_id", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_item_id", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_class_id", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_class", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_category", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_category_id", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_brand", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_units", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_rec_end_date", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_color", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] }, { "Name": "i_product_name", "LFTags": [{ "CatalogId": "123456789111", "TagKey": "usergroup", "TagValues": [ "developer" ] }] } ] }] }
For more information, see Viewing the resources that a LF-Tag is assigned to
in the Amazon Lake Formation Developer Guide. -
For API details, see SearchTablesByLfTags
in Amazon CLI Command Reference.
-
The following code example shows how to use start-query-planning
.
- Amazon CLI
-
To process query statement
The following
start-query-planning
example submits a request to process a query statement.aws lakeformation start-query-planning \ --cli-input-json
file://input.json
Contents of
input.json
:{ "QueryPlanningContext": { "CatalogId": "012345678901", "DatabaseName": "tpc" }, "QueryString": "select * from dl_tpc_household_demographics_gov where hd_income_band_sk=9" }
Output:
{ "QueryId": "772a273f-4a62-4cda-8d98-69615ee8be9b" }
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see StartQueryPlanning
in Amazon CLI Command Reference.
-
The following code example shows how to use start-transaction
.
- Amazon CLI
-
To start new transaction
The following
start-transaction
example starts a new transaction and returns its transaction ID.aws lakeformation start-transaction \ --transaction-type
=
'READ_AND_WRITE
'Output:
{ "TransactionId": "b014d972ca8347b89825e33c5774aec4" }
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see StartTransaction
in Amazon CLI Command Reference.
-
The following code example shows how to use update-lf-tag
.
- Amazon CLI
-
To update LF-Tag definition
The following
update-lf-tag
example updates LF-Tag definition.aws lakeformation update-lf-tag \ --catalog-id '
123456789111
' \ --tag-key 'usergroup
' \ --tag-values-to-add '["admin"]
'This command produces no output.
For more information, see Managing LF-Tags for metadata access control
in the Amazon Lake Formation Developer Guide. -
For API details, see UpdateLfTag
in Amazon CLI Command Reference.
-
The following code example shows how to use update-table-objects
.
- Amazon CLI
-
To modify objects of governed table
The following
update-table-objects
example adds provided S3 objects to the specified governed table.aws lakeformation update-table-objects \ --cli-input-json
file://input.json
Contents of
input.json
:{ "CatalogId": "012345678901", "DatabaseName": "tpc", "TableName": "dl_tpc_household_demographics_gov", "TransactionId": "12347a9f75424b9b915f6ff201d2a190", "WriteOperations": [{ "AddObject": { "Uri": "s3://lf-data-lake-012345678901/target/dl_tpc_household_demographics_gov/run-unnamed-1-part-block-0-r-00000-snappy-ff26b17504414fe88b302cd795eabd00.parquet", "ETag": "1234ab1fc50a316b149b4e1f21a73800", "Size": 42200 } }] }
This command produces no output.
For more information, see Reading from and writing to the data lake within transactions
in the Amazon Lake Formation Developer Guide. -
For API details, see UpdateTableObjects
in Amazon CLI Command Reference.
-