Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Key capabilities

The capabilities below are available to any Amazon EMR Spark application that uses the EvaluateDataQuality transform. All of them behave the same on Amazon EMR as on Amazon Glue; the sections that follow summarize each one and link to the detailed reference in the Amazon Glue Developer Guide.

Rule evaluation and DQDL

You author rulesets in DQDL, the declarative language that Amazon EMR Data Quality uses to describe data quality expectations. The transform supports more than 30 built-in rule types spanning completeness, uniqueness, value constraints, aggregations, freshness, schema shape, and pattern matching, and lets you combine them with boolean operators, thresholds, and with clauses that filter the rows a rule applies to. The grammar is identical on Amazon EMR and on Amazon Glue.

For the full rule catalog, parameter reference, and worked examples, see Data Quality Definition Language (DQDL) reference in the Amazon Glue Developer Guide.

Analyzers

Analyzers compute statistics about your dataset without asserting a pass/fail outcome. They are useful for exploring a new dataset, for seeding thresholds on a rule, and for training the anomaly-detection model. You declare them in the Analyzers block of a ruleset.

Rules = [
    RowCount > 1000
]

Analyzers = [
    RowCount,
    Completeness "customer_id",
    Uniqueness "customer_id",
    DistinctValuesCount "country",
    Mean "order_total"
]

Each analyzer value is written to the results store as a time series, keyed by the EvaluationContext, and is available to the anomaly-detection model. The same set of analyzers is supported on Amazon EMR and on Amazon Glue.

For the full analyzer catalog and parameter reference, see Analyzers in the Amazon Glue Developer Guide.

Anomaly detection

Anomaly detection learns the trend and seasonality of a statistic from its own history and flags runs where the observed value falls outside the expected range, without you having to re-tune thresholds by hand. Every time your Amazon EMR Spark application runs Data Quality with publishing enabled, the values of each rule metric and each analyzer statistic are appended to a per-statistic time series, keyed by EvaluationContext. The model trains on that time series, predicts the expected range for the next run, and emits an observation when the observed value falls outside that range.

To enable anomaly detection on an Amazon EMR run, set observations.scope to ALL in additional_options, and keep enableDataQualityResultsPublishing set to true. The observations.scope option tells the transform which analyzers in your ruleset produce observations; ALL scores every published statistic.

For the detection algorithm, mode selection, feedback workflow, and recommendation types, see Anomaly detection in the Amazon Glue Developer Guide. The anomaly-detection engine and algorithm are shared between Amazon EMR Data Quality and Amazon Glue Data Quality. For Amazon EMR runs, the time series of statistics and the anomaly observations are visible in the Amazon Glue console on the Data quality tab of the Amazon Glue Data Catalog table only when your application sets publishToDataCatalogTarget. Without catalog attribution, retrieve the statistic time series through ListDataQualityStatistics and the anomaly observations through ListDataQualityStatisticAnnotations.

Dynamic rules

Dynamic rules compare the current value of a statistic to the history of that statistic for previous runs of the same EvaluationContext. This lets you write rules such as "row count should be within 10 percent of last run" without hard-coding a threshold. Unlike anomaly detection, which learns expected ranges from history automatically, dynamic rules require you to define the comparison logic explicitly in your ruleset. Dynamic rules require that publishing was enabled on prior runs so that the history exists.

Rules = [
    RowCount between last(1) * 0.9 and last(1) * 1.1,
    Mean "order_total" between avg(last(7)) - 2 * std(last(7))
                          and avg(last(7)) + 2 * std(last(7))
]

For the full dynamic-rule syntax and supported functions, see Dynamic rules in the Amazon Glue Developer Guide.