FlagDuplicatesInColumn class - Amazon Glue
ExampleOutput — methods —__call__applynamedescribeArgsdescribeReturndescribeTransformdescribeErrorsdescribe
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

FlagDuplicatesInColumn class

The FlagDuplicatesInColumn transform returns a new column with a specified value in each row that indicates whether the value in the row's source column matches a value in an earlier row of the source column. When matches are found, they are flagged as duplicates. The initial occurrence is not flagged, because it doesn't match an earlier row.

Example

from pyspark.context import SparkContext
from pyspark.sql import SparkSession      
from awsgluedi.transforms import *

sc = SparkContext()
spark = SparkSession(sc)

datasource1 = spark.read.json("s3://${BUCKET}/json/zips/raw/data")

try:
    df_output = column.FlagDuplicatesInColumn.apply(
        data_frame=datasource1,
        spark_context=sc,
        source_column="city",
        target_column="flag_col",
        true_string="True",
        false_string="False"
    )
except:
    print("Unexpected Error happened ")
    raise

Output

The FlagDuplicatesInColumn transformation will add a new column `flag_col` to the `df_output` DataFrame. This column will contain a string value indicating whether the corresponding row has a duplicate value in the `city` column or not. If a row has a duplicate `city` value, the `flag_col` will contain the `true_string` value "True". If a row has a unique `city` value, the `flag_col` will contain the `false_string` value "False".

The resulting `df_output` DataFrame will contain all columns from the original `datasource1` DataFrame, plus the additional `flag_col` column indicating duplicate `city` values.

Methods

__call__(spark_context, data_frame, source_column, target_column, true_string=DEFAULT_TRUE_STRING, false_string=DEFAULT_FALSE_STRING)

The FlagDuplicatesInColumn transform returns a new column with a specified value in each row that indicates whether the value in the row's source column matches a value in an earlier row of the source column. When matches are found, they are flagged as duplicates. The initial occurrence is not flagged, because it doesn't match an earlier row.

  • source_column – Name of the source column.

  • target_column – Name of the target column.

  • true_string – String to be inserted in the target column when a source column value duplicates an earlier value in that column.

  • false_string – String to be inserted in the target column when a source column value is distinct from earlier values in that column.

apply(cls, *args, **kwargs)

Inherited from GlueTransform apply.

name(cls)

Inherited from GlueTransform name.

describeArgs(cls)

Inherited from GlueTransform describeArgs.

describeReturn(cls)

Inherited from GlueTransform describeReturn.

describeTransform(cls)

Inherited from GlueTransform describeTransform.

describeErrors(cls)

Inherited from GlueTransform describeErrors.

describe(cls)

Inherited from GlueTransform describe.