FlatMap 类
对集合中的每个 DynamicFrame 应用转换。结果不会拼凑成单个 DynamicFrame，而是作为一个集合保存。
FlatMap 的示例
以下示例片段展示了当应用于 FlatMap 时，如何对动态帧集合使用 ResolveChoice 转换。用于输入的数据是位于占位符 Amazon S3 地址 s3://bucket/path-for-data/sample.json 的 JSON，包含以下数据。
[{ "firstname": "Arnav", "lastname": "Desai", "address": { "street": "6 Anyroad Avenue", "city": "London", "state": "England", "country": "UK" }, "phone": 17235550101, "affiliations": [ "General Anonymous Example Products", "Example Independent Research", "Government Department of Examples" ] }, { "firstname": "Mary", "lastname": "Major", "address": { "street": "7821 Spot Place", "city": "Centerville", "state": "OK", "country": "US" }, "phone": 19185550023, "affiliations": [ "Example Dot Com", "Example Independent Research", "Example.io" ] }, { "firstname": "Paulo", "lastname": "Santos", "address": { "street": "123 Maple Street", "city": "London", "state": "Ontario", "country": "CA" }, "phone": 12175550181, "affiliations": [ "General Anonymous Example Products", "Example Dot Com" ] }]
例 将 ResolveChoice 应用于 DynamicFrameCollection，将显示输出。
#Read DynamicFrame datasource = glueContext.create_dynamic_frame_from_options("s3", connection_options = {"paths":["s3://bucket/path/to/file/mysamplejson.json"]}, format="json") datasource.printSchema() datasource.show() ## Split to create a DynamicFrameCollection split_frame=datasource.split_fields(["firstname","lastname","address"],"personal_info","business_info") split_frame.keys() print("---") ## Use FlatMap to run ResolveChoice kwargs = {"choice": "cast:string"} flat = FlatMap.apply(split_frame, ResolveChoice, frame_name="frame", transformation_ctx='tcx', **kwargs) flat.keys() ##Select one of the DynamicFrames personal_info = flat.select("personal_info") personal_info.printSchema() personal_info.show() print("---") business_info = flat.select("business_info") business_info.printSchema() business_info.show()
重要
调用 FlatMap.apply 时，frame_name 参数必须为 "frame"。目前不接受其他值。
root |-- firstname: string |-- lastname: string |-- address: struct | |-- street: string | |-- city: string | |-- state: string | |-- country: string |-- phone: long |-- affiliations: array | |-- element: string --- { "firstname": "Mary", "lastname": "Major", "address": { "street": "7821 Spot Place", "city": "Centerville", "state": "OK", "country": "US" }, "phone": 19185550023, "affiliations": [ "Example Dot Com", "Example Independent Research", "Example.io" ] } { "firstname": "Paulo", "lastname": "Santos", "address": { "street": "123 Maple Street", "city": "London", "state": "Ontario", "country": "CA" }, "phone": 12175550181, "affiliations": [ "General Anonymous Example Products", "Example Dot Com" ] } --- root |-- firstname: string |-- lastname: string |-- address: struct | |-- street: string | |-- city: string | |-- state: string | |-- country: string { "firstname": "Mary", "lastname": "Major", "address": { "street": "7821 Spot Place", "city": "Centerville", "state": "OK", "country": "US" } } { "firstname": "Paulo", "lastname": "Santos", "address": { "street": "123 Maple Street", "city": "London", "state": "Ontario", "country": "CA" } } --- root |-- phone: long |-- affiliations: array | |-- element: string { "phone": 19185550023, "affiliations": [ "Example Dot Com", "Example Independent Research", "Example.io" ] } { "phone": 12175550181, "affiliations": [ "General Anonymous Example Products", "Example Dot Com" ] }
方法
frame_name __call__ (dfc，BaseTransform transformation_ctx = “”，**，base_kwargs)，
向集合中的每个 DynamicFrame 应用转换，并展平结果。
dfc– 要展平的DynamicFrameCollection(必需)。BaseTransform– 派生自GlueTransform、要应用于集合的每个成员的转换 (必需)。frame_name– 要将集合元素传递到的参数名称 (必需)。transformation_ctx– 用于标识状态信息的唯一字符串 (可选)。base_kwargs– 要传递到基本转换的参数 (必需)。
通过将返回新 DynamicFrameCollection 创建的每个转换 DynamicFrame 在源 DynamicFrameCollection。
apply(cls, *args, **kwargs)
继承自 GlueTransform apply。
name(cls)
继承自 GlueTransform name。
describeArgs(cls)
继承自 GlueTransform describeArgs。
describeReturn(cls)
继承自 GlueTransform describeReturn。
describeTransform(cls)
继承自 GlueTransform describeTransform。
describeErrors(cls)
继承自 GlueTransform describeErrors。
describe(cls)
继承自 GlueTransform describe。