Working with expired items - Amazon DynamoDB
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Working with expired items

Expired items that are pending deletion can be filtered from read and write operations. This is useful in scenarios when expired data is no longer valid and should not be used. If they are not filtered, they’ll continue to show in read and write operations until they are deleted by the background process.

Note

These items still count towards storage and read costs until they are deleted.

TTL deletions can be identified in DynamoDB Streams, but only in the Region where the deletion occurred. TTL deletions that are replicated to global table regions are not identifiable in DynamoDB streams in the regions the deletion is replicated to.

Filter expired items from read operations

For read operations such as Scan and Query, a filter expression can filter out expired items that are pending deletion. As shown in the code snippet below, the filter expression can filter out items where the TTL time is equal to or less than the current time. This is done with an assignment statement that obtains the current time as a variable (now), which is converted to int for epoch time format.

Python
import boto3 from datetime import datetime def query_dynamodb_items(table_name, partition_key): """ :param table_name: Name of the DynamoDB table :param partition_key: :return: """ try: # Initialize a DynamoDB resource dynamodb = boto3.resource('dynamodb', region_name='us-east-1') # Specify your table table = dynamodb.Table(table_name) # Get the current time in epoch format current_time = int(datetime.now().timestamp()) # Perform the query operation with a filter expression to exclude expired items # response = table.query( # KeyConditionExpression=boto3.dynamodb.conditions.Key('partitionKey').eq(partition_key), # FilterExpression=boto3.dynamodb.conditions.Attr('expireAt').gt(current_time) # ) response = table.query( KeyConditionExpression=dynamodb.conditions.Key('partitionKey').eq(partition_key), FilterExpression=dynamodb.conditions.Attr('expireAt').gt(current_time) ) # Print the items that are not expired for item in response['Items']: print(item) except Exception as e: print(f"Error querying items: {e}") # Call the function with your values query_dynamodb_items('Music', 'your-partition-key-value')

The output from the update operation shows that, while the createdAt time is unchanged, the updatedAt and expireAt times have been updated. The expireAt time is now set to 90 days from the time of the last update, which is Thursday, October 19, 2023 at 1:27:15 PM.

partition_key createdAt updatedAt expireAt attribute_1 attribute_2

some_value

2023-07-17 14:11:05.322323 2023-07-19 13:27:15.213423 1697722035 new_value some_value
Javascript
import { DynamoDBClient, QueryCommand } from "@aws-sdk/client-dynamodb"; import { marshall, unmarshall } from "@aws-sdk/util-dynamodb"; async function queryDynamoDBItems(tableName, region, primaryKey) { const client = new DynamoDBClient({ region: region, endpoint: `https://dynamodb.${region}.amazonaws.com` }); const currentTime = Math.floor(Date.now() / 1000); const params = { TableName: tableName, KeyConditionExpression: "#pk = :pk", FilterExpression: "#ea > :ea", ExpressionAttributeNames: { "#pk": "primaryKey", "#ea": "expireAt" }, ExpressionAttributeValues: marshall({ ":pk": primaryKey, ":ea": currentTime }) }; try { const { Items } = await client.send(new QueryCommand(params)); Items.forEach(item => { console.log(unmarshall(item)) }); return Items; } catch (err) { console.error(`Error querying items: ${err}`); throw err; } } //enter your own values here queryDynamoDBItems('your-table-name', 'your-partition-key-value');

Conditionally write to expired items

A condition expression can be used to avoid writes against expired items. The code snippet below is a conditional update that checks whether the expiration time is greater than the current time. If true, the write operation will continue.

Python
import boto3 from datetime import datetime, timedelta from botocore.exceptions import ClientError def update_dynamodb_item(table_name, region, primary_key, sort_key, ttl_attribute): """ Updates an existing record in a DynamoDB table with a new or updated TTL attribute. :param table_name: Name of the DynamoDB table :param region: AWS Region of the table - example `us-east-1` :param primary_key: one attribute known as the partition key. :param sort_key: Also known as a range attribute. :param ttl_attribute: name of the TTL attribute in the target DynamoDB table :return: """ try: dynamodb = boto3.resource('dynamodb', region_name=region) table = dynamodb.Table(table_name) # Generate updated TTL in epoch second format updated_expiration_time = int((datetime.now() + timedelta(days=90)).timestamp()) # Define the update expression for adding/updating a new attribute update_expression = "SET newAttribute = :val1" # Define the condition expression for checking if 'ttlExpirationDate' is not expired condition_expression = "ttlExpirationDate > :val2" # Define the expression attribute values expression_attribute_values = { ':val1': ttl_attribute, ':val2': updated_expiration_time } response = table.update_item( Key={ 'primaryKey': primary_key, 'sortKey': sort_key }, UpdateExpression=update_expression, ConditionExpression=condition_expression, ExpressionAttributeValues=expression_attribute_values ) print("Item updated successfully.") return response['ResponseMetadata']['HTTPStatusCode'] # Ideally a 200 OK except ClientError as e: if e.response['Error']['Code'] == "ConditionalCheckFailedException": print("Condition check failed: Item's 'ttlExpirationDate' is expired.") else: print(f"Error updating item: {e}") except Exception as e: print(f"Error updating item: {e}") # replace with your values update_dynamodb_item('your-table-name', 'us-east-1', 'your-partition-key-value', 'your-sort-key-value', 'your-ttl-attribute-value')

The output from the update operation shows that, while the createdAt time is unchanged, the updatedAt and expireAt times have been updated. The expireAt time is now set to 90 days from the time of the last update, which is Thursday, October 19, 2023 at 1:27:15 PM.

partition_key createdAt updatedAt expireAt attribute_1 attribute_2

some_value

2023-07-17 14:11:05.322323 2023-07-19 13:27:15.213423 1697722035 new_value some_value
Javascript
import { DynamoDBClient, UpdateItemCommand } from "@aws-sdk/client-dynamodb"; import { marshall, unmarshall } from "@aws-sdk/util-dynamodb"; // Example function to update an item in a DynamoDB table. // The function should take the table name, region, partition key, sort key, and new attribute as arguments. // The function should use the DynamoDB client to update the item. // The function should return the updated item. // The function should handle errors and exceptions. const updateDynamoDBItem = async (tableName, region, partitionKey, sortKey, newAttribute) => { const client = new DynamoDBClient({ region: region, endpoint: `https://dynamodb.${region}.amazonaws.com` }); const currentTime = Math.floor(Date.now() / 1000); const params = { TableName: tableName, Key: marshall({ artist: partitionKey, album: sortKey }), UpdateExpression: "SET newAttribute = :newAttribute", ConditionExpression: "expireAt > :expiration", ExpressionAttributeValues: marshall({ ':newAttribute': newAttribute, ':expiration': currentTime }), ReturnValues: "ALL_NEW" }; try { const response = await client.send(new UpdateItemCommand(params)); const responseData = unmarshall(response.Attributes); console.log("Item updated successfully: ", responseData); return responseData; } catch (error) { if (error.name === "ConditionalCheckFailedException") { console.log("Condition check failed: Item's 'expireAt' is expired."); } else { console.error("Error updating item: ", error); } throw error; } }; // Enter your values here updateDynamoDBItem('your-table-name', "us-east-1",'your-partition-key-value', 'your-sort-key-value', 'your-new-attribute-value');

Identifying deleted items in DynamoDB Streams

The streams record contains a user identity field Records[<index>].userIdentity. Items that are deleted by the TTL process have the following fields:

Records[<index>].userIdentity.type "Service" Records[<index>].userIdentity.principalId "dynamodb.amazonaws.com"

The following JSON shows the relevant portion of a single streams record:

"Records": [ { ... "userIdentity": { "type": "Service", "principalId": "dynamodb.amazonaws.com" } ... } ]