本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。
在 Amazon Redshift 中进行读取和写入
以下代码示例用于使用数据源和使用 PySpark Spark 从包含数据源的 Amazon Redshift 数据库中读取API和写入示例数据。SQL
- Data source API
-
PySpark 用于从带有数据源的 Amazon Redshift 数据库中读取和写入示例数据。API
import boto3 from pyspark.sql import SQLContext sc = # existing SparkContext sql_context = SQLContext(sc) url = "jdbc:redshift:iam://redshifthost:5439/database" aws_iam_role_arn = "arn:aws:iam::
accountID
:role/roleName
" df = sql_context.read \ .format("io.github.spark_redshift_community.spark.redshift") \ .option("url",url
) \ .option("dbtable", "tableName
") \ .option("tempdir", "s3://path/for/temp/data
") \ .option("aws_iam_role", "aws_iam_role_arn
") \ .load() df.write \ .format("io.github.spark_redshift_community.spark.redshift") \ .option("url",url
) \ .option("dbtable", "tableName_copy
") \ .option("tempdir", "s3://path/for/temp/data
") \ .option("aws_iam_role", "aws_iam_role_arn
") \ .mode("error") \ .save() - SparkSQL
-
PySpark 用于通过 Spark 读取和写入亚马逊 Redshift 数据库的示例数据。SQL
import boto3 import json import sys import os from pyspark.sql import SparkSession spark = SparkSession \ .builder \ .enableHiveSupport() \ .getOrCreate() url = "jdbc:redshift:iam://redshifthost:5439/database" aws_iam_role_arn = "arn:aws:iam::
accountID
:role/roleName
" bucket = "s3://path/for/temp/data
" tableName = "tableName
" # Redshift table name s = f"""CREATE TABLE IF NOT EXISTS {tableName
} (country string, data string) USING io.github.spark_redshift_community.spark.redshift OPTIONS (dbtable '{tableName
}', tempdir '{bucket
}', url '{url
}', aws_iam_role '{aws_iam_role_arn
}' ); """ spark.sql(s) columns = ["country" ,"data"] data = [("test-country
","test-data
")] df = spark.sparkContext.parallelize(data).toDF(columns) # Insert data into table df.write.insertInto(tableName
, overwrite=False) df = spark.sql(f"SELECT * FROM {tableName
}") df.show()
对 Amazon Redshift 进行身份验证
注意事项