Troubleshooting data quality monitoring at the catalog and database level
Use this topic to find out why a data quality evaluation configuration doesn't evaluate a table, why its evaluation runs fail, or why a request to create or update a configuration fails.
Contents
Resolving missing evaluation runs for a table
Evaluation configurations work in the background. If Amazon Glue Data Quality can't evaluate a table before an evaluation run starts, for example because the role lacks a permission, no run appears for that table. If you don't see runs or results for a table, check the following causes in order.
-
The evaluation hasn't started yet – A
ONCEevaluation starts about a minute after you create the configuration, and evaluations for new tables and metadata changes typically start within a few minutes. When your account is near its quota of concurrent data quality evaluation runs, evaluations wait until capacity is available. -
The configuration isn't set to evaluate now – Call
GetDataQualityEvaluationConfigand check thatPauseRunsisfalse, that the triggers match what you expect, and, for aSCHEDULEtrigger, that the current time is betweenStartAtandEndAt. Schedule expressions use UTC. -
The table isn't in scope – Check that the table is in the configuration's catalog and database and that its name matches
TableNamePattern. Also check whether a more specific configuration applies to the table. A table-level or database-level configuration takes precedence over a broader one, even when it is paused or uses different triggers. For more information, see When configurations overlap. -
The table was evaluated less than an hour ago – Each table is evaluated at most once an hour across all of your configurations. A trigger that fires within an hour of the table's previous automatic evaluation skips the table.
-
The table type isn't supported – Configurations skip tables that use a JDBC connection, views other than Amazon Athena views, and multi-dialect views. For more information, see Supported tables.
-
The trigger doesn't apply to this change – The
METADATA_CHANGEtrigger responds to changes to the table in the Amazon Glue Data Catalog. Writing files to Amazon S3 without updating the table, and adding partitions, don't trigger an evaluation. -
The role can't be assumed or lacks permissions – Permissions can change after you create the configuration. Check that the role's trust policy still allows Amazon Glue to assume it, and that its permissions policy includes the actions in Permissions policy for the configuration role. In particular, check
glue:SearchTables,glue:GetTable,glue:StartDataQualityRulesetEvaluationRun, andiam:PassRoleon the role itself. If you use Lake Formation, check that the role has theDESCRIBEpermission on the table. -
The ruleset no longer exists – If you deleted the ruleset that the configuration's
RulesetNamerefers to, recreate it or update the configuration to use another ruleset.
To see the calls that Amazon Glue makes on your behalf, look in Amazon CloudTrail for events from the
configuration's role, such as AssumeRole, SearchTables,
GetTable, and StartDataQualityRulesetEvaluationRun, and check them
for AccessDenied errors. When Amazon Glue finds the tables in scope and starts their
evaluation runs, the role session name begins with glue-dq-autorun-.
Resolving failed evaluation runs
When an evaluation run starts but fails, it appears with the FAILED status
in the list of evaluation runs for the table. To see why, call
GetDataQualityRulesetEvaluationRun and read ErrorString. Common
causes include the following:
-
A rule references a column that the table doesn't have – A ruleset in a catalog-level or database-level configuration runs against every table in scope. Use rules that apply to any table, or move table-specific rules to a table-level configuration.
-
The preprocessing query doesn't work for every table – A
PreProcessingQueryin a broad configuration must be valid for each table in scope. Use the{{DATABASE_NAME}}and{{TABLE_NAME}}placeholders, and filter only on columns that every table has. -
The role can't read the data – Check that the role can read the table's data location in Amazon S3, and decrypt it if it uses an Amazon KMS key.
-
Lake Formation denies access to the data – If the error says
FULL SELECT privileges required on the table, grant the role theSELECTpermission on the table without column or data filters. If it saysInsufficient Lake Formation permissions. Verify the data lake settings, allow external engines to access data with full table access in the Lake Formation data lake settings.
Resolving errors when you create or update a configuration
Amazon Glue validates a configuration when you create or update it. The following table lists common errors and how to resolve them.
| Error message | Resolution |
|---|---|
Unable to assume role for arn: |
Update the role's trust policy to allow glue.amazonaws.com to assume
it, with conditions that match your account and the catalog and table ARNs. For an
example, see Trust policy for the configuration role. |
Data quality evaluation configuration role |
Add glue:GetCatalog, glue:GetDatabase, or
glue:GetTable to the role's permissions policy, depending on the
configuration's level. |
... does not have the Lake Formation DESCRIBE permission on the
|
Grant the DESCRIBE permission in Lake Formation on the catalog, database, or
table. The message names the principal that needs it: the configuration's role or the
caller. |
User: |
Allow the caller to pass the configuration's role to Amazon Glue. For more information, see Permissions to manage evaluation configurations. |
CatalogId must be in the calling account or Cross-account
pass role is not allowed. |
Create the configuration in the account that owns the catalog, with a role from the same account. |
Cannot find Data Quality Ruleset in account |
Create the ruleset with CreateDataQualityRuleset in the same account
and Region before you create the configuration, or check the ruleset name. |
ScheduleExpression can run at most hourly, so minutes must be a single
value |
Use a single value in the minutes field of the cron expression, such as
cron(0 * * * ? *). For rate expressions, use at least one hour. |
ScheduleExpression has no future run times |
Choose an expression and a StartAt and EndAt window
that include at least one future run. |
PauseRuns can only be set when Triggers include SCHEDULE or
METADATA_CHANGE |
A configuration with only the ONCE trigger can't be paused. To stop
it, delete it. |
Role is required unless the Update only sets PauseRuns |
Include the role in the update request. |
Specify either RulesetName or ManagedRulesetName, not both |
Choose a built-in ruleset or your own ruleset. |
RowLevelResults requires a RulesetName ... |
Built-in rulesets contain only analyzers, which don't produce rule results or row-level results. Remove these run options, or use your own ruleset. |
Data quality evaluation configuration count ( |
Delete configurations that you no longer need, or combine them by using database-level or catalog-level configurations. |