Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions,
see Getting Started with Amazon Web Services in China
(PDF).
Working with expired items and time to live (TTL)
Expired items that are pending deletion can be filtered from read and write operations.
This is useful in scenarios when expired data is no longer valid and should not be used. If
they are not filtered, they’ll continue to show in read and write operations until they are
deleted by the background process.
These items still count towards storage and read costs until they are deleted.
TTL deletions can be identified in DynamoDB Streams, but only in the Region where the
deletion occurred. TTL deletions that are replicated to global table regions are not
identifiable in DynamoDB streams in the regions the deletion is replicated to.
Filter expired items from read operations
For read operations such as Scan and Query, a filter expression can filter out expired items that are pending deletion. As shown in the following code snippet, the filter expression can filter out items where the TTL time is equal to or less than the current time. For example, the Python SDK code includes an assignment statement that obtains the current time as a variable (now
), and converts it into int
for epoch time format.
The following code examples show how to query for TTL items.
- Java
-
- SDK for Java 2.x
-
Query Filtered Expression to gather TTL items in a DynamoDB table.
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.services.dynamodb.model.QueryRequest;
import software.amazon.awssdk.services.dynamodb.model.QueryResponse;
import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
import software.amazon.awssdk.utils.ImmutableMap;
import java.util.Map;
import java.util.Optional;
// Get current time in epoch second format (comparing against expiry attribute)
final long currentTime = System.currentTimeMillis() / 1000;
// A string that contains conditions that DynamoDB applies after the Query operation, but before the data is returned to you.
final String keyConditionExpression = "#pk = :pk";
// The condition that specifies the key values for items to be retrieved by the Query action.
final String filterExpression = "#ea > :ea";
final Map<String, String> expressionAttributeNames = ImmutableMap.of(
"#pk", "primaryKey",
"#ea", "expireAt");
final Map<String, AttributeValue> expressionAttributeValues = ImmutableMap.of(
":pk", AttributeValue.builder().s(primaryKey).build(),
":ea", AttributeValue.builder().s(String.valueOf(currentTime)).build()
);
final QueryRequest request = QueryRequest.builder()
.tableName(tableName)
.keyConditionExpression(keyConditionExpression)
.filterExpression(filterExpression)
.expressionAttributeNames(expressionAttributeNames)
.expressionAttributeValues(expressionAttributeValues)
.build();
try (DynamoDbClient ddb = DynamoDbClient.builder()
.region(region)
.build()) {
final QueryResponse response = ddb.query(request);
System.out.println(tableName + " Query operation with TTL successful. Request id is "
+ response.responseMetadata().requestId());
// Print the items that are not expired
for (Map<String, AttributeValue> item : response.items()) {
System.out.println(item.toString());
}
} catch (ResourceNotFoundException e) {
System.err.format("Error: The Amazon DynamoDB table \"%s\" can't be found.\n", tableName);
System.exit(1);
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
System.exit(0);
- JavaScript
-
- SDK for JavaScript (v3)
-
import { DynamoDBClient, QueryCommand } from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
async function queryDynamoDBItems(tableName, region, primaryKey) {
const client = new DynamoDBClient({
region: region,
endpoint: `https://dynamodb.${region}.amazonaws.com`
});
const currentTime = Math.floor(Date.now() / 1000);
const params = {
TableName: tableName,
KeyConditionExpression: "#pk = :pk",
FilterExpression: "#ea > :ea",
ExpressionAttributeNames: {
"#pk": "primaryKey",
"#ea": "expireAt"
},
ExpressionAttributeValues: marshall({
":pk": primaryKey,
":ea": currentTime
})
};
try {
const { Items } = await client.send(new QueryCommand(params));
Items.forEach(item => {
console.log(unmarshall(item))
});
return Items;
} catch (err) {
console.error(`Error querying items: ${err}`);
throw err;
}
}
//enter your own values here
queryDynamoDBItems('your-table-name', 'your-partition-key-value');
- Python
-
- SDK for Python (Boto3)
-
import boto3
from datetime import datetime
def query_dynamodb_items(table_name, partition_key):
"""
:param table_name: Name of the DynamoDB table
:param partition_key:
:return:
"""
try:
# Initialize a DynamoDB resource
dynamodb = boto3.resource('dynamodb',
region_name='us-east-1')
# Specify your table
table = dynamodb.Table(table_name)
# Get the current time in epoch format
current_time = int(datetime.now().timestamp())
# Perform the query operation with a filter expression to exclude expired items
# response = table.query(
# KeyConditionExpression=boto3.dynamodb.conditions.Key('partitionKey').eq(partition_key),
# FilterExpression=boto3.dynamodb.conditions.Attr('expireAt').gt(current_time)
# )
response = table.query(
KeyConditionExpression=dynamodb.conditions.Key('partitionKey').eq(partition_key),
FilterExpression=dynamodb.conditions.Attr('expireAt').gt(current_time)
)
# Print the items that are not expired
for item in response['Items']:
print(item)
except Exception as e:
print(f"Error querying items: {e}")
# Call the function with your values
query_dynamodb_items('Music', 'your-partition-key-value')
Conditionally write to expired items
A condition expression can be used to avoid writes against expired items. The code snippet below is a conditional update that checks whether the expiration time is greater than the current time. If true, the write operation will continue.
The following code examples show how to conditionally update an item's TTL.
- Java
-
- SDK for Java 2.x
-
package com.amazon.samplelib.ttl;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
import software.amazon.awssdk.services.dynamodb.model.AttributeValue;
import software.amazon.awssdk.services.dynamodb.model.DynamoDbException;
import software.amazon.awssdk.services.dynamodb.model.ResourceNotFoundException;
import software.amazon.awssdk.services.dynamodb.model.UpdateItemRequest;
import software.amazon.awssdk.services.dynamodb.model.UpdateItemResponse;
import software.amazon.awssdk.utils.ImmutableMap;
import java.util.Map;
import java.util.Optional;
public class UpdateTTLConditional {
public static void main(String[] args) {
final String usage = """
Usage:
<tableName> <primaryKey> <sortKey> <newTtlAttribute> <region>
Where:
tableName - The Amazon DynamoDB table being queried.
primaryKey - The name of the primary key. Also known as the hash or partition key.
sortKey - The name of the sort key. Also known as the range attribute.
newTtlAttribute - New attribute name (as part of the update command)
region (optional) - The AWS region that the Amazon DynamoDB table is located in. (Default: us-east-1)
""";
// Optional "region" parameter - if args list length is NOT 3 or 4, short-circuit exit.
if (!(args.length == 4 || args.length == 5)) {
System.out.println(usage);
System.exit(1);
}
final String tableName = args[0];
final String primaryKey = args[1];
final String sortKey = args[2];
final String newTtlAttribute = args[3];
Region region = Optional.ofNullable(args[4]).isEmpty() ? Region.US_EAST_1 : Region.of(args[4]);
// Get current time in epoch second format
final long currentTime = System.currentTimeMillis() / 1000;
// Calculate expiration time 90 days from now in epoch second format
final long expireDate = currentTime + (90 * 24 * 60 * 60);
// An expression that defines one or more attributes to be updated, the action to be performed on them, and new values for them.
final String updateExpression = "SET newTtlAttribute = :val1";
// A condition that must be satisfied in order for a conditional update to succeed.
final String conditionExpression = "expireAt > :val2";
final ImmutableMap<String, AttributeValue> keyMap =
ImmutableMap.of("primaryKey", AttributeValue.fromS(primaryKey),
"sortKey", AttributeValue.fromS(sortKey));
final Map<String, AttributeValue> expressionAttributeValues = ImmutableMap.of(
":val1", AttributeValue.builder().s(newTtlAttribute).build(),
":val2", AttributeValue.builder().s(String.valueOf(expireDate)).build()
);
final UpdateItemRequest request = UpdateItemRequest.builder()
.tableName(tableName)
.key(keyMap)
.updateExpression(updateExpression)
.conditionExpression(conditionExpression)
.expressionAttributeValues(expressionAttributeValues)
.build();
try (DynamoDbClient ddb = DynamoDbClient.builder()
.region(region)
.build()) {
final UpdateItemResponse response = ddb.updateItem(request);
System.out.println(tableName + " UpdateItem operation with conditional TTL successful. Request id is "
+ response.responseMetadata().requestId());
} catch (ResourceNotFoundException e) {
System.err.format("Error: The Amazon DynamoDB table \"%s\" can't be found.\n", tableName);
System.exit(1);
} catch (DynamoDbException e) {
System.err.println(e.getMessage());
System.exit(1);
}
System.exit(0);
}
}
- JavaScript
-
- SDK for JavaScript (v3)
-
Update TTL on on an existing DynamoDB Item in a table, with a condition.
import { DynamoDBClient, UpdateItemCommand } from "@aws-sdk/client-dynamodb";
import { marshall, unmarshall } from "@aws-sdk/util-dynamodb";
const updateDynamoDBItem = async (tableName, region, partitionKey, sortKey, newAttribute) => {
const client = new DynamoDBClient({
region: region,
endpoint: `https://dynamodb.${region}.amazonaws.com`
});
const currentTime = Math.floor(Date.now() / 1000);
const params = {
TableName: tableName,
Key: marshall({
artist: partitionKey,
album: sortKey
}),
UpdateExpression: "SET newAttribute = :newAttribute",
ConditionExpression: "expireAt > :expiration",
ExpressionAttributeValues: marshall({
':newAttribute': newAttribute,
':expiration': currentTime
}),
ReturnValues: "ALL_NEW"
};
try {
const response = await client.send(new UpdateItemCommand(params));
const responseData = unmarshall(response.Attributes);
console.log("Item updated successfully: ", responseData);
return responseData;
} catch (error) {
if (error.name === "ConditionalCheckFailedException") {
console.log("Condition check failed: Item's 'expireAt' is expired.");
} else {
console.error("Error updating item: ", error);
}
throw error;
}
};
// Enter your values here
updateDynamoDBItem('your-table-name', "us-east-1",'your-partition-key-value', 'your-sort-key-value', 'your-new-attribute-value');
- Python
-
- SDK for Python (Boto3)
-
import boto3
from datetime import datetime, timedelta
from botocore.exceptions import ClientError
def update_dynamodb_item(table_name, region, primary_key, sort_key, ttl_attribute):
"""
Updates an existing record in a DynamoDB table with a new or updated TTL attribute.
:param table_name: Name of the DynamoDB table
:param region: AWS Region of the table - example `us-east-1`
:param primary_key: one attribute known as the partition key.
:param sort_key: Also known as a range attribute.
:param ttl_attribute: name of the TTL attribute in the target DynamoDB table
:return:
"""
try:
dynamodb = boto3.resource('dynamodb', region_name=region)
table = dynamodb.Table(table_name)
# Generate updated TTL in epoch second format
updated_expiration_time = int((datetime.now() + timedelta(days=90)).timestamp())
# Define the update expression for adding/updating a new attribute
update_expression = "SET newAttribute = :val1"
# Define the condition expression for checking if 'expireAt' is not expired
condition_expression = "expireAt > :val2"
# Define the expression attribute values
expression_attribute_values = {
':val1': ttl_attribute,
':val2': updated_expiration_time
}
response = table.update_item(
Key={
'primaryKey': primary_key,
'sortKey': sort_key
},
UpdateExpression=update_expression,
ConditionExpression=condition_expression,
ExpressionAttributeValues=expression_attribute_values
)
print("Item updated successfully.")
return response['ResponseMetadata']['HTTPStatusCode'] # Ideally a 200 OK
except ClientError as e:
if e.response['Error']['Code'] == "ConditionalCheckFailedException":
print("Condition check failed: Item's 'expireAt' is expired.")
else:
print(f"Error updating item: {e}")
except Exception as e:
print(f"Error updating item: {e}")
# replace with your values
update_dynamodb_item('your-table-name', 'us-east-1', 'your-partition-key-value', 'your-sort-key-value',
'your-ttl-attribute-value')
Identifying deleted items in DynamoDB
Streams
The streams record contains a user identity field
Records[<index>].userIdentity
. Items that are deleted by the TTL
process have the following fields:
Records[<index>].userIdentity.type
"Service"
Records[<index>].userIdentity.principalId
"dynamodb.amazonaws.com"
The following JSON shows the relevant portion of a single streams record:
"Records": [
{
...
"userIdentity": {
"type": "Service",
"principalId": "dynamodb.amazonaws.com"
}
...
}
]