将 Apache Iceberg 与 Amazon EMR on EKS 结合使用

将 Apache Iceberg 与 Amazon EMR on EKS 应用程序结合使用

1. 启动任务运行以在应用程序配置中提交 Spark 任务时，请包含 Iceberg Spark 运行时 JAR 文件：

   --job-driver '{"sparkSubmitJobDriver" : {"sparkSubmitParameters" : "--jars local:///usr/share/aws/iceberg/lib/iceberg-spark3-runtime.jar"}}'

2. 包含 Iceberg 额外配置：

   --configuration-overrides '{
       "applicationConfiguration": [
       "classification" : "spark-defaults", 
       "properties" : {
           "spark.sql.catalog.dev.warehouse" : "s3://DOC-EXAMPLE-BUCKET/EXAMPLE-PREFIX/ ", 
           "spark.sql.extensions ":" org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions ", 
           "spark.sql.catalog.dev" : "org.apache.iceberg.spark.SparkCatalog",
           "spark.sql.catalog.dev.catalog-impl" : "org.apache.iceberg.aws.glue.GlueCatalog",
           "spark.sql.catalog.dev.io-impl": "org.apache.iceberg.aws.s3.S3FileIO"
           }
       ]
   }'