View a markdown version of this page

Getting started with data quality monitoring at the catalog and database level - Amazon Glue
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Getting started with data quality monitoring at the catalog and database level

Data quality monitoring at the catalog and database level evaluates every table in a Amazon Glue Data Catalog catalog or database from a single data quality evaluation configuration, including tables that you add later. To learn how configurations choose tables, triggers, and rulesets, see Data quality monitoring at the catalog and database level.

In this topic, you create an IAM role, create a configuration for a database with the Amazon CLI, and then find its results. Each CLI command has a matching operation in the Amazon SDKs. In the examples, replace the values in italics with your own, and replace 111122223333 with your account ID.

Prerequisites

Before you begin, make sure that you have the following:

  • One or more tables in the Amazon Glue Data Catalog with data in Amazon S3 or Amazon S3 Tables.

  • The latest version of the Amazon CLI or the Amazon SDK that you use, so that it includes the data quality evaluation configuration operations.

  • Permissions to manage evaluation configurations, to pass the configuration's IAM role, and, if you use Lake Formation, the Lake Formation permissions on the catalog, database, or table. For more information about these permissions, see Permissions to manage evaluation configurations.

Step 1: Create an IAM role for the configuration

Amazon Glue Data Quality assumes the configuration's role to find the tables in scope, start an evaluation run for each table, and read the table's data during the run. The role needs a trust policy that allows Amazon Glue to assume it, and a permissions policy for these tasks.

To create the configuration role
  1. Open a terminal where the Amazon CLI is configured for the account that owns your tables.

  2. Save the trust policy from Trust policy for the configuration role as trust-policy.json, replacing the Region and account ID with your own.

  3. Save the permissions policy from Permissions policy for the configuration role as permissions-policy.json, replacing the Region, account ID, role name, and Amazon S3 bucket with your own.

  4. Create the role. The examples in this topic use the role name DataQualityEvaluationConfigRole. If you choose another name, replace it in the commands and policies.

    aws iam create-role \ --role-name DataQualityEvaluationConfigRole \ --assume-role-policy-document file://trust-policy.json
  5. Attach the permissions policy to the role. Replace DataQualityEvaluationConfigPolicy with a name for the policy.

    aws iam put-role-policy \ --role-name DataQualityEvaluationConfigRole \ --policy-name DataQualityEvaluationConfigPolicy \ --policy-document file://permissions-policy.json
  6. If you use Lake Formation, grant the role the Lake Formation permissions in Permissions policy for the configuration role. The configuration evaluates only the tables that its role can describe.

Step 2: Create a data quality evaluation configuration

The following command creates a configuration at the database level that gathers baseline statistics for every table in a database. It uses the default settings: the SIMPLE built-in ruleset and the ONCE trigger.

aws glue create-data-quality-evaluation-config \ --name my-database-config \ --level DATABASE \ --data-source '{ "GlueTables": { "CatalogId": "111122223333", "DatabaseName": "my_database" } }' \ --role DataQualityEvaluationConfigRole

The response returns the name of the configuration.

{ "Name": "my-database-config", "EvaluationRunMode": "EVALUATION_CONFIG" }

When you create a configuration, Amazon Glue checks that the catalog, database, or table exists, that your role can be assumed and can describe it, and that the ruleset, if you specify one, exists. About a minute later, the ONCE trigger starts an evaluation run for each table in the database. Because this configuration includes ONCE, tables that you add to the database later are each evaluated once when they are created.

To confirm the settings that the configuration uses, call get-data-quality-evaluation-config.

aws glue get-data-quality-evaluation-config --name my-database-config
{ "Name": "my-database-config", "EvaluationRunMode": "EVALUATION_CONFIG", "CreatedOn": "2026-11-16T17:02:11.512000+00:00", "LastModifiedOn": "2026-11-16T17:02:11.512000+00:00", "Level": "DATABASE", "DataSource": { "GlueTables": { "CatalogId": "111122223333", "DatabaseName": "my_database" } }, "ManagedRulesetName": "SIMPLE", "Triggers": [ "ONCE" ], "PauseRuns": false, "Role": "arn:aws:iam::111122223333:role/DataQualityEvaluationConfigRole" }

Amazon Glue returns the role as a full ARN, even if you specified a role name.

Step 3: View the results

Each evaluation stores its results with the table. To list the results that your configuration produced for a table, filter list-data-quality-results on the table, the evaluation run mode, and the configuration name.

aws glue list-data-quality-results \ --filter '{ "DataSource": { "GlueTable": { "DatabaseName": "my_database", "TableName": "my_table" } }, "EvaluationRunMode": "EVALUATION_CONFIG", "DataQualityEvaluationConfig": "my-database-config" }'

Each result in the response includes EvaluationRunMode and DataQualityEvaluationConfig, so you can tell results from your configurations apart from results of runs that you started yourself.

{ "Results": [ { "ResultId": "dqresult-a1b2c3d4e5f6EXAMPLE", "DataSource": { "GlueTable": { "DatabaseName": "my_database", "TableName": "my_table", "CatalogId": "111122223333" } }, "StartedOn": "2026-11-16T17:04:38.204000+00:00", "EvaluationRunMode": "EVALUATION_CONFIG", "DataQualityEvaluationConfig": "my-database-config" } ] }

A built-in ruleset gathers statistics rather than evaluating rules. To view the statistics, get the result's profile ID and list its statistics. In the second command, replace profile-id with the profile ID that the first command returns.

aws glue get-data-quality-result \ --result-id dqresult-a1b2c3d4e5f6EXAMPLE \ --query ProfileId --output text aws glue list-data-quality-statistics \ --profile-id profile-id

You can also see each run in the list of evaluation runs for the table, including runs that failed.

aws glue list-data-quality-ruleset-evaluation-runs \ --filter '{ "DataSource": { "GlueTable": { "DatabaseName": "my_database", "TableName": "my_table" } } }'

If no results appear after a few minutes, see Troubleshooting data quality monitoring at the catalog and database level.

More configuration examples

The following examples show configurations for other levels, triggers, and rulesets. Replace the values in italics with your own, and replace 111122223333 with your account ID.

Evaluate a whole catalog on a schedule

The following configuration profiles every table in the account's default catalog every 24 hours, starting on December 1, 2026. A rate expression, such as rate(24 hours), runs at a fixed interval. Because new tables are always evaluated, tables that you add to the catalog are also profiled when they are created.

aws glue create-data-quality-evaluation-config \ --name my-catalog-config \ --level CATALOG \ --data-source '{"GlueTables": {"CatalogId": "111122223333"}}' \ --managed-ruleset-name SIMPLE \ --triggers SCHEDULE \ --schedule-expression "rate(24 hours)" \ --start-at 2026-12-01T00:00:00Z \ --role DataQualityEvaluationConfigRole

To run at a specific time of day instead, use a cron expression. For example, cron(0 6 * * ? *) runs every day at 6:00 AM UTC.

Evaluate tables that match a name pattern

The following configuration evaluates only the tables in a database whose names start with a prefix, every 12 hours, and whenever their metadata changes. It uses the ADVANCED built-in ruleset, which gathers more statistics for anomaly detection to learn from.

aws glue create-data-quality-evaluation-config \ --name my-pattern-config \ --level DATABASE \ --data-source '{ "GlueTables": { "CatalogId": "111122223333", "DatabaseName": "my_other_database", "TableNamePattern": "my_prefix_*" } }' \ --managed-ruleset-name ADVANCED \ --triggers SCHEDULE METADATA_CHANGE \ --schedule-expression "rate(12 hours)" \ --role DataQualityEvaluationConfigRole

Evaluate a table with your own rules

To hold a table to specific standards, create a ruleset with create-data-quality-ruleset, and then create a table-level configuration that uses it. For a TABLE configuration, set TableNamePattern to the table name. This configuration takes precedence over the database configuration from Step 2 for this table. It runs at the start of every hour.

aws glue create-data-quality-ruleset \ --name my-ruleset \ --ruleset 'Rules = [ RowCount > 0, IsComplete "id_column", IsUnique "id_column" ]' aws glue create-data-quality-evaluation-config \ --name my-table-config \ --level TABLE \ --data-source '{ "GlueTables": { "CatalogId": "111122223333", "DatabaseName": "my_database", "TableNamePattern": "my_table" } }' \ --ruleset-name my-ruleset \ --triggers SCHEDULE \ --schedule-expression "cron(0 * * * ? *)" \ --role DataQualityEvaluationConfigRole

Evaluate recent data only

To evaluate a subset of each table, add a PreProcessingQuery to the data source. In catalog-level and database-level configurations, use the {{DATABASE_NAME}} and {{TABLE_NAME}} placeholders so that the query reads each table in scope. The following configuration evaluates the last seven days of data in each table of a database every day at 2:00 AM UTC. Each table in scope must have the column that the query filters on.

aws glue create-data-quality-evaluation-config \ --name my-recent-data-config \ --level DATABASE \ --data-source '{ "GlueTables": { "CatalogId": "111122223333", "DatabaseName": "my_database", "PreProcessingQuery": "SELECT * FROM `{{DATABASE_NAME}}.{{TABLE_NAME}}` WHERE date_column >= date_sub(current_date(), 7)" } }' \ --triggers SCHEDULE \ --schedule-expression "cron(0 2 * * ? *)" \ --role DataQualityEvaluationConfigRole

Evaluate tables in Amazon S3 Tables

For Amazon S3 Tables, set CatalogId to the Amazon S3 Tables catalog for your table bucket, and DatabaseName to the namespace. The following configuration profiles every table in a namespace every Monday at 3:00 AM UTC. You and the role also need Amazon S3 Tables permissions. For more information about these permissions, see Permissions to manage evaluation configurations and Permissions policy for the configuration role.

aws glue create-data-quality-evaluation-config \ --name my-s3-tables-config \ --level DATABASE \ --data-source '{ "GlueTables": { "CatalogId": "111122223333:s3tablescatalog/amzn-s3-demo-table-bucket", "DatabaseName": "my_namespace" } }' \ --triggers SCHEDULE \ --schedule-expression "cron(0 3 ? * MON *)" \ --role DataQualityEvaluationConfigRole

Managing evaluation configurations

List configurations

To list the configurations in your account and Region, call list-data-quality-evaluation-configs. You can filter by level.

aws glue list-data-quality-evaluation-configs --level DATABASE

Update a configuration

Call update-data-quality-evaluation-config with the configuration name, the role, and the settings to change. Settings that you don't specify keep their current values. The following command changes the configuration from Step 2 to the ADVANCED ruleset and adds a schedule that runs every 24 hours.

aws glue update-data-quality-evaluation-config \ --name my-database-config \ --managed-ruleset-name ADVANCED \ --triggers ONCE SCHEDULE \ --schedule-expression "rate(24 hours)" \ --role DataQualityEvaluationConfigRole

Keep the following in mind when you update a configuration:

  • You must specify the role in every update, except an update that only sets PauseRuns.

  • You can't change the name, the level, or the catalog of a configuration. You can change the database and the table name pattern within the same level. To change the level or catalog, create a new configuration.

  • A list of triggers replaces the current list. If the new list doesn't include SCHEDULE, the schedule expression and time window are removed.

  • DataSource, AdditionalRunOptions, and AdditionalDataSources replace their current values as a whole. If you specify a DataSource without a PreProcessingQuery, the configuration no longer uses a preprocessing query.

  • Specifying RulesetName replaces a built-in ruleset, and specifying ManagedRulesetName replaces your own ruleset.

  • Keeping ONCE in the triggers doesn't evaluate the tables again. Adding ONCE to a configuration that didn't have it evaluates every table in scope one more time.

  • Evaluation runs that have already started aren't affected. Evaluations that haven't started yet use the updated settings.

  • To add or remove tags, use tag-resource and untag-resource with the configuration's ARN.

Pause and resume a configuration

To pause a configuration that uses the SCHEDULE or METADATA_CHANGE trigger, set PauseRuns to true. While it is paused, the configuration evaluates nothing, including new tables. In the Amazon CLI, PauseRuns is a Boolean flag: --pause-runs sets it to true, and --no-pause-runs sets it to false. The CLI doesn't accept a value after the flag.

aws glue update-data-quality-evaluation-config \ --name my-database-config \ --pause-runs

To resume, set PauseRuns to false. The configuration continues with its existing triggers, and evaluations that were missed while it was paused aren't run later.

aws glue update-data-quality-evaluation-config \ --name my-database-config \ --no-pause-runs

Delete a configuration

Deleting a configuration stops all future evaluations from it. Evaluation runs that have already started finish, and existing results remain with each table. The built-in rulesets that Amazon Glue created in your account aren't deleted, because other configurations can use them.

aws glue delete-data-quality-evaluation-config --name my-database-config

Example: Manage configurations with the Amazon SDK for Python (Boto3)

The following Python example creates a configuration at the database level, waits for its evaluations to produce results, and prints the results for one table. Replace the configuration, database, table, and role names, and the account ID, with your own values.

import logging import time import boto3 from botocore.exceptions import ClientError logger = logging.getLogger(__name__) glue = boto3.client("glue") try: glue.create_data_quality_evaluation_config( Name="my-database-config", Level="DATABASE", DataSource={ "GlueTables": { "CatalogId": "111122223333", "DatabaseName": "my_database", } }, ManagedRulesetName="SIMPLE", Triggers=["ONCE"], Role="DataQualityEvaluationConfigRole", ) except ClientError as err: logger.error( "Couldn't create the data quality evaluation configuration. Here's why: %s: %s", err.response["Error"]["Code"], err.response["Error"]["Message"]) raise result_filter = { "DataSource": {"GlueTable": {"DatabaseName": "my_database", "TableName": "my_table"}}, "EvaluationRunMode": "EVALUATION_CONFIG", "DataQualityEvaluationConfig": "my-database-config", } results = [] for _ in range(30): results = glue.list_data_quality_results(Filter=result_filter)["Results"] if results: break time.sleep(60) for summary in results: result = glue.get_data_quality_result(ResultId=summary["ResultId"]) statistics = glue.list_data_quality_statistics(ProfileId=result["ProfileId"]) for statistic in statistics["Statistics"]: print(statistic["StatisticName"], statistic.get("ColumnsReferenced"), statistic.get("DoubleValue"))