AWS CloudFormation
User Guide (API Version 2010-05-15)
AWS 服务或AWS文档中描述的功能,可能因地区/位置而异。请点击 Amazon AWS 入门,可查看中国地区的具体差异

AWS::DataPipeline::Pipeline

创建可用于自动处理数据的移动和转换的数据管道。在每个管道中,您可定义管道对象,例如活动、计划、数据节点和资源。有关可使用的管道对象和组件的信息,请参阅 AWS Data Pipeline 开发人员指南 中的管道对象引用

语法

要在 AWS CloudFormation 模板中声明此实体,请使用以下语法:

JSON

Copy
{ "Type" : "AWS::DataPipeline::Pipeline", "Properties" : { "Activate" : Boolean, "Description" : String, "Name" : String, "ParameterObjects" : [ Parameter object, ... ], "ParameterValues" : [ Parameter value, ... ], "PipelineObjects" : [ Pipeline object, ... ], "PipelineTags" : [ Pipeline tag, ... ] } }

YAML

Copy
Type: "AWS::DataPipeline::Pipeline" Properties: Activate: Boolean Description: String Name: String ParameterObjects: - Parameter object ParameterValues: - Parameter value PipelineObjects: - Pipeline object PipelineTags: - Pipeline tag

属性

Activate

指示是否验证并启动管道或停止活动管道。默认情况下, 值设为 true.

Required: No

Type: Boolean

更新要求无需中断

Description

管道的描述。

Required: No

Type: String

更新要求替换.

Name

管道的名称。由于 AWS CloudFormation 为每个新管道分配一个唯一标识符,您可使用与您的 AWS 账户关联的多个管道的相同名称。

Required: Yes

Type: String

更新要求替换

ParameterObjects

定义位于管道定义中的变量。有关更多信息,请参阅 AWS Data Pipeline 开发人员指南 中的使用参数化模板创建管道

Required: No

类型AWS Data Pipeline 管道 ParameterObjects

更新要求无需中断

ParameterValues

定义在 ParameterObjects 属性中定义的参数的值。有关更多信息,请参阅 AWS Data Pipeline 开发人员指南 中的使用参数化模板创建管道

Required: No

类型AWS Data Pipeline 管道 ParameterValues

更新要求无需中断

PipelineObjects

组成管道的管道对象的列表。有关管道对象和每个对象的描述的更多信息,请参阅 AWS Data Pipeline 开发人员指南 中的管道对象引用

Required: Yes

类型AWS Data Pipeline PipelineObject 的列表。

更新要求时而中断. 并不能更新所有对象、字段和值。有关可更新内容的限制已记录在 AWS Data Pipeline 开发人员指南 中的编辑您的管道中。

PipelineTags

与管道关联的任意标签(键/值对)的列表,该列表可用于控制权限。有关更多信息,请参阅 AWS Data Pipeline 开发人员指南 中的控制对管道和资源的访问

Required: No

类型AWS Data Pipeline 管道 PipelineTags

更新要求无需中断

返回值

Ref

当该资源的逻辑 ID 提供给 Ref内部函数时,Ref 将返回资源名称。

当您指定 AWS::DataPipeline::Pipeline 资源作为 Ref 函数的参数时,AWS CloudFormation 会返回管道 ID。

有关使用 Ref 功能的更多信息,请参阅参考

示例

下面的数据管道将来自 Amazon DynamoDB (DynamoDB) 表的数据备份到 Amazon Simple Storage Service (Amazon S3) 存储桶。管道使用 HiveCopyActivity 活动复制数据并每天运行一次。管道的角色和管道资源在相同模板中的其他位置声明。

JSON

Copy
"DynamoDBInputS3OutputHive": { "Type": "AWS::DataPipeline::Pipeline", "Properties": { "Name": "DynamoDBInputS3OutputHive", "Description": "Pipeline to backup DynamoDB data to S3", "Activate": "true", "ParameterObjects": [ { "Id": "myDDBReadThroughputRatio", "Attributes": [ { "Key": "description", "StringValue": "DynamoDB read throughput ratio" }, { "Key": "type", "StringValue": "Double" }, { "Key": "default", "StringValue": "0.2" } ] }, { "Id": "myOutputS3Loc", "Attributes": [ { "Key": "description", "StringValue": "S3 output bucket" }, { "Key": "type", "StringValue": "AWS::S3::ObjectKey" }, { "Key": "default", "StringValue": { "Fn::Join" : [ "", [ "s3://", { "Ref": "S3OutputLoc" } ] ] } } ] }, { "Id": "myDDBTableName", "Attributes": [ { "Key": "description", "StringValue": "DynamoDB Table Name " }, { "Key": "type", "StringValue": "String" } ] } ], "ParameterValues": [ { "Id": "myDDBTableName", "StringValue": { "Ref": "TableName" } } ], "PipelineObjects": [ { "Id": "S3BackupLocation", "Name": "Copy data to this S3 location", "Fields": [ { "Key": "type", "StringValue": "S3DataNode" }, { "Key": "dataFormat", "RefValue": "DDBExportFormat" }, { "Key": "directoryPath", "StringValue": "#{myOutputS3Loc}/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}" } ] }, { "Id": "DDBSourceTable", "Name": "DDBSourceTable", "Fields": [ { "Key": "tableName", "StringValue": "#{myDDBTableName}" }, { "Key": "type", "StringValue": "DynamoDBDataNode" }, { "Key": "dataFormat", "RefValue": "DDBExportFormat" }, { "Key": "readThroughputPercent", "StringValue": "#{myDDBReadThroughputRatio}" } ] }, { "Id": "DDBExportFormat", "Name": "DDBExportFormat", "Fields": [ { "Key": "type", "StringValue": "DynamoDBExportDataFormat" } ] }, { "Id": "TableBackupActivity", "Name": "TableBackupActivity", "Fields": [ { "Key": "resizeClusterBeforeRunning", "StringValue": "true" }, { "Key": "type", "StringValue": "HiveCopyActivity" }, { "Key": "input", "RefValue": "DDBSourceTable" }, { "Key": "runsOn", "RefValue": "EmrClusterForBackup" }, { "Key": "output", "RefValue": "S3BackupLocation" } ] }, { "Id": "DefaultSchedule", "Name": "RunOnce", "Fields": [ { "Key": "occurrences", "StringValue": "1" }, { "Key": "startAt", "StringValue": "FIRST_ACTIVATION_DATE_TIME" }, { "Key": "type", "StringValue": "Schedule" }, { "Key": "period", "StringValue": "1 Day" } ] }, { "Id": "Default", "Name": "Default", "Fields": [ { "Key": "type", "StringValue": "Default" }, { "Key": "scheduleType", "StringValue": "cron" }, { "Key": "failureAndRerunMode", "StringValue": "CASCADE" }, { "Key": "role", "StringValue": "DataPipelineDefaultRole" }, { "Key": "resourceRole", "StringValue": "DataPipelineDefaultResourceRole" }, { "Key": "schedule", "RefValue": "DefaultSchedule" } ] }, { "Id": "EmrClusterForBackup", "Name": "EmrClusterForBackup", "Fields": [ { "Key": "terminateAfter", "StringValue": "2 Hours" }, { "Key": "amiVersion", "StringValue": "3.3.2" }, { "Key": "masterInstanceType", "StringValue": "m1.medium" }, { "Key": "coreInstanceType", "StringValue": "m1.medium" }, { "Key": "coreInstanceCount", "StringValue": "1" }, { "Key": "type", "StringValue": "EmrCluster" } ] } ] } }

YAML

Copy
DynamoDBInputS3OutputHive: Type: "AWS::DataPipeline::Pipeline" Properties: Name: DynamoDBInputS3OutputHive Description: "Pipeline to backup DynamoDB data to S3" Activate: true ParameterObjects: - Id: "myDDBReadThroughputRatio" Attributes: - Key: "description" StringValue: "DynamoDB read throughput ratio" - Key: "type" StringValue: "Double" - Key: "default" StringValue: "0.2" - Id: "myOutputS3Loc" Attributes: - Key: "description" StringValue: "S3 output bucket" - Key: "type" StringValue: "AWS::S3::ObjectKey" - Key: "default" StringValue: Fn::Join: - "" - - "s3://" - Ref: "S3OutputLoc" - Id: "myDDBTableName" Attributes: - Key: "description" StringValue: "DynamoDB Table Name " - Key: "type" StringValue: "String" ParameterValues: - Id: "myDDBTableName" StringValue: Ref: "TableName" PipelineObjects: - Id: "S3BackupLocation" Name: "Copy data to this S3 location" Fields: - Key: "type" StringValue: "S3DataNode" - Key: "dataFormat" RefValue: "DDBExportFormat" - Key: "directoryPath" StringValue: "#{myOutputS3Loc}/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}" - Id: "DDBSourceTable" Name: "DDBSourceTable" Fields: - Key: "tableName" StringValue: "#{myDDBTableName}" - Key: "type" StringValue: "DynamoDBDataNode" - Key: "dataFormat" RefValue: "DDBExportFormat" - Key: "readThroughputPercent" StringValue: "#{myDDBReadThroughputRatio}" - Id: "DDBExportFormat" Name: "DDBExportFormat" Fields: - Key: "type" StringValue: "DynamoDBExportDataFormat" - Id: "TableBackupActivity" Name: "TableBackupActivity" Fields: - Key: "resizeClusterBeforeRunning" StringValue: "true" - Key: "type" StringValue: "HiveCopyActivity" - Key: "input" RefValue: "DDBSourceTable" - Key: "runsOn" RefValue: "EmrClusterForBackup" - Key: "output" RefValue: "S3BackupLocation" - Id: "DefaultSchedule" Name: "RunOnce" Fields: - Key: "occurrences" StringValue: "1" - Key: "startAt" StringValue: "FIRST_ACTIVATION_DATE_TIME" - Key: "type" StringValue: "Schedule" - Key: "period" StringValue: "1 Day" - Id: "Default" Name: "Default" Fields: - Key: "type" StringValue: "Default" - Key: "scheduleType" StringValue: "cron" - Key: "failureAndRerunMode" StringValue: "CASCADE" - Key: "role" StringValue: "DataPipelineDefaultRole" - Key: "resourceRole" StringValue: "DataPipelineDefaultResourceRole" - Key: "schedule" RefValue: "DefaultSchedule" - Id: "EmrClusterForBackup" Name: "EmrClusterForBackup" Fields: - Key: "terminateAfter" StringValue: "2 Hours" - Key: "amiVersion" StringValue: "3.3.2" - Key: "masterInstanceType" StringValue: "m1.medium" - Key: "coreInstanceType" StringValue: "m1.medium" - Key: "coreInstanceCount" StringValue: "1" - Key: "type" StringValue: "EmrCluster"

本页内容: