Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Migrating from GlueContext/Glue DynamicFrame to Spark DataFrame.

The following are Python and Scala examples of migrating GlueContext/Glue DynamicFrame in Glue 4.0 to Spark DataFrame in Glue 5.0.

Python

Before:

escaped_table_name= '`<dbname>`.`<table_name>`'

additional_options = {
  "query": f'select * from {escaped_table_name} WHERE column1 = 1 AND column7 = 7'
}

# DynamicFrame example
dataset = glueContext.create_data_frame_from_catalog(
    database="<dbname>",
    table_name=escaped_table_name, 
    additional_options=additional_options)

After:

table_identifier= '`<catalogname>`.`<dbname>`.`<table_name>`"' #catalogname is optional

# DataFrame example
dataset = spark.sql(f'select * from {table_identifier} WHERE column1 = 1 AND column7 = 7')

Scala

Before:

val escapedTableName = "`<dbname>`.`<table_name>`"

val additionalOptions = JsonOptions(Map(
    "query" -> s"select * from $escapedTableName WHERE column1 = 1 AND column7 = 7"
    )
)

# DynamicFrame example
val datasource0 = glueContext.getCatalogSource(
    database="<dbname>", 
    tableName=escapedTableName, 
    additionalOptions=additionalOptions).getDataFrame()

After:

val tableIdentifier = "`<catalogname>`.`<dbname>`.`<table_name>`" //catalogname is optional

# DataFrame example
val datasource0 = spark.sql(s"select * from $tableIdentifier WHERE column1 = 1 AND column7 = 7")