# Amazon EMR Data Quality
<a name="emr-eks-data-quality"></a>

Amazon EMR Data Quality, powered by [Amazon Glue Data Quality](https://docs.amazonaws.cn/glue/latest/dg/glue-data-quality.html), lets you measure and monitor the quality of the data that your Amazon EMR on EKS Spark workloads read and write without leaving the Spark application. This page covers the Amazon EMR on EKS prerequisites and walkthrough. For concepts, APIs, monitoring, and troubleshooting, see [Amazon EMR Data Quality](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-data-quality.html) in the *Amazon EMR Management Guide*.

## Prerequisites
<a name="emr-eks-dq-prerequisites"></a>

Amazon EMR Data Quality is installed by default in Amazon EMR releases emr-spark-8.1.0 and later. Only Apache Spark workloads are supported.

The IAM job execution role that your Amazon EMR on EKS Spark application assumes (passed via `--execution-role-arn` on `start-job-run`) must grant the following permissions to run Data Quality and publish results.


| Action | When required | 
| --- | --- | 
| glue:PublishDataQuality | To publish each evaluation to Amazon Glue Data Quality so you can view history, compare runs, and feed anomaly detection. Publishing is on by default (enableDataQualityResultsPublishing is True), so grant this permission unless you explicitly opt out by setting the option to False. | 
| glue:AccessDataQualityRuntimeConfiguration | To fetch publishing configuration from the service when the library initializes publishing. Required whenever publishing is enabled; without it, the library silently disables publishing for the run. | 
| cloudwatch:PutMetricData | To emit per-run pass and fail metrics to Amazon CloudWatch for dashboards and alarms. Required only when you set enableDataQualityCloudWatchMetrics to True. | 
| glue:GetTable, glue:GetDatabase | To read catalog table metadata. Required when the input DataFrame is read from a Amazon Glue Data Catalog table, or when the application passes publishToDataCatalogTarget so the service can attribute the run to a specific catalog table. | 
| glue:GetDataQualityRuleset | To fetch a saved ruleset from the Amazon Glue Data Catalog. Required only when your Spark script loads a ruleset from the catalog instead of defining DQDL inline. See [Fetching a saved ruleset from the Amazon Glue Data Catalog](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-dq-getting-started.html#emr-dq-saved-rulesets). | 

The following inline policy grants the minimum set of permissions required to publish Data Quality results from an Amazon EMR on EKS Spark application.

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "GlueDataQuality",
            "Effect": "Allow",
            "Action": [
                "glue:PublishDataQuality",
                "glue:AccessDataQualityRuntimeConfiguration"
            ],
            "Resource": "*"
        }
    ]
}
```

## Run your first evaluation
<a name="emr-eks-dq-walkthrough"></a>

1. **Register an Amazon EMR on EKS virtual cluster.** Use release `emr-spark-8.1.0` or later. Register a virtual cluster against your EKS cluster namespace and attach a job execution role that has the permissions listed in [Prerequisites](#emr-eks-dq-prerequisites). See [Setting up Amazon EMR on EKS](https://docs.amazonaws.cn/emr/latest/EMR-on-EKS-DevelopmentGuide/setting-up.html) for the full prerequisites.

   For example, to create a virtual cluster, run:

   ```
   aws emr-containers create-virtual-cluster \
   --name My_Spark_DataQuality_VirtualCluster \
   --container-provider '{
       "id": "{{<eks-cluster-name>}}",
       "type": "EKS",
       "info": { "eksInfo": { "namespace": "{{<namespace>}}" } }
   }'
   ```

1. **Define a ruleset and evaluate it.** The following PySpark script defines a ruleset in DQDL, evaluates it against a DataFrame, and prints the per-rule outcome.

   ```
   from pyspark.sql import SparkSession
   from awsdataquality.transforms import EvaluateDataQuality
   
   spark = SparkSession.builder.appName("dq-example").getOrCreate()
   
   df = spark.read.parquet("s3://{{amzn-s3-demo-bucket}}/orders/")
   
   ruleset = """
   Rules = [
       RowCount > 1000,
       IsComplete "order_id",
       IsUnique "order_id",
       ColumnValues "status" in ["PLACED", "SHIPPED", "CANCELED"],
       Mean "order_total" between 10 and 500
   ]
   """
   
   results = EvaluateDataQuality.apply(
       frame=df,
       ruleset=ruleset,
       publishing_options={
           "dataQualityEvaluationContext": "orders_daily",
           "enableDataQualityResultsPublishing": True,
       },
   )
   
   results.show(truncate=False)
   ```

   The `results` DataFrame contains one row per rule with the following columns: `Rule`, `Outcome`, `FailureReason`, and `EvaluatedMetrics`. Rows with an `Outcome` of `Failed` include the observed value in `FailureReason`.

1. **Submit the job run.** Submit the script as a job run against your virtual cluster.

   ```
   aws emr-containers start-job-run \
   --virtual-cluster-id {{<virtual-cluster-id>}} \
   --name dq-example \
   --execution-role-arn {{<execution-role-arn>}} \
   --release-label emr-spark-8.1.0 \
   --job-driver '{
       "sparkSubmitJobDriver": {
           "entryPoint": "s3://{{amzn-s3-demo-bucket}}/scripts/dq_example.py"
       }
   }'
   ```

1. **Retrieve the result.** After the job run finishes, list the run and fetch its detail with the Amazon Glue Data Quality APIs or the Amazon CLI. If your application also sets `publishToDataCatalogTarget` with a `DatabaseName` and `TableName`, the run appears on the *Data quality* tab of that catalog table in the Amazon Glue console.

   ```
   aws glue list-data-quality-results-v2 --max-results 5
   
   aws glue get-data-quality-result-v2 --result-id {{<result-id>}}
   ```

## Learn more about Amazon EMR Data Quality
<a name="emr-eks-dq-learn-more"></a>

For the complete reference, see the following topics in the *Amazon EMR Management Guide*:
+ [Amazon EMR Data Quality overview, concepts, and limits](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-data-quality.html)
+ [Key capabilities (DQDL, analyzers, anomaly detection, dynamic rules)](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-dq-key-capabilities.html)
+ [Fetching a saved ruleset from the Amazon Glue Data Catalog](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-dq-getting-started.html#emr-dq-saved-rulesets)
+ [Working with results](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-dq-working-with-results.html)
+ [Monitoring and alerting with Amazon CloudWatch](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-dq-monitoring.html)
+ [Troubleshooting](https://docs.amazonaws.cn/emr/latest/ManagementGuide/emr-dq-troubleshooting.html)