This is the new *Amazon CloudFormation Template Reference Guide*. Please update your bookmarks and links. For help getting started with CloudFormation, see the [Amazon CloudFormation User Guide](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/Welcome.html).
# AWS::DataPipeline::Pipeline
The AWS::DataPipeline::Pipeline resource specifies a data pipeline that you can use to automate the movement and transformation of data.
**Important**
Amazon Data Pipeline is no longer available to new customers. Existing customers of Amazon Data Pipeline can continue to use the service as normal. [Learn more](https://amazonaws-china.com/blogs/big-data/migrate-workloads-from-aws-data-pipeline/)
In each pipeline, you define pipeline objects, such as activities, schedules, data nodes, and resources.
The `AWS::DataPipeline::Pipeline` resource adds tasks, schedules, and preconditions to the specified pipeline. You can use `PutPipelineDefinition` to populate a new pipeline.
`PutPipelineDefinition` also validates the configuration as it adds it to the pipeline. Changes to the pipeline are saved unless one of the following validation errors exist in the pipeline.
+ An object is missing a name or identifier field.
+ A string or reference field is empty.
+ The number of objects in the pipeline exceeds the allowed maximum number of objects.
+ The pipeline is in a FINISHED state.
Pipeline object definitions are passed to the [PutPipelineDefinition](https://docs.amazonaws.cn/datapipeline/latest/APIReference/API_PutPipelineDefinition.html) action and returned by the [GetPipelineDefinition](https://docs.amazonaws.cn/datapipeline/latest/APIReference/API_GetPipelineDefinition.html) action.
## Syntax
To declare this entity in your Amazon CloudFormation template, use the following syntax:
### JSON
```
{
"Type" : "AWS::DataPipeline::Pipeline",
"Properties" : {
"[Activate](#cfn-datapipeline-pipeline-activate)" : {{Boolean}},
"[Description](#cfn-datapipeline-pipeline-description)" : {{String}},
"[Name](#cfn-datapipeline-pipeline-name)" : {{String}},
"[ParameterObjects](#cfn-datapipeline-pipeline-parameterobjects)" : {{[ ParameterObject, ... ]}},
"[ParameterValues](#cfn-datapipeline-pipeline-parametervalues)" : {{[ ParameterValue, ... ]}},
"[PipelineObjects](#cfn-datapipeline-pipeline-pipelineobjects)" : {{[ PipelineObject, ... ]}},
"[PipelineTags](#cfn-datapipeline-pipeline-pipelinetags)" : {{[ PipelineTag, ... ]}}
}
}
```
### YAML
```
Type: AWS::DataPipeline::Pipeline
Properties:
[Activate](#cfn-datapipeline-pipeline-activate): {{Boolean}}
[Description](#cfn-datapipeline-pipeline-description): {{String}}
[Name](#cfn-datapipeline-pipeline-name): {{String}}
[ParameterObjects](#cfn-datapipeline-pipeline-parameterobjects): {{
- ParameterObject}}
[ParameterValues](#cfn-datapipeline-pipeline-parametervalues): {{
- ParameterValue}}
[PipelineObjects](#cfn-datapipeline-pipeline-pipelineobjects): {{
- PipelineObject}}
[PipelineTags](#cfn-datapipeline-pipeline-pipelinetags): {{
- PipelineTag}}
```
## Properties
`Activate`
Indicates whether to validate and start the pipeline or stop an active pipeline. By default, the value is set to `true`.
*Required*: No
*Type*: Boolean
*Update requires*: [No interruption](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-no-interrupt)
`Description`
A description of the pipeline.
*Required*: No
*Type*: String
*Pattern*: `[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\r\n\t]*`
*Minimum*: `0`
*Maximum*: `1024`
*Update requires*: [Replacement](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-replacement)
`Name`
The name of the pipeline.
*Required*: Yes
*Type*: String
*Pattern*: `[\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF\n\t]*`
*Minimum*: `1`
*Maximum*: `1024`
*Update requires*: [Replacement](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-replacement)
`ParameterObjects`
The parameter objects used with the pipeline.
*Required*: No
*Type*: Array of [ParameterObject](aws-properties-datapipeline-pipeline-parameterobject.md)
*Update requires*: [No interruption](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-no-interrupt)
`ParameterValues`
The parameter values used with the pipeline.
*Required*: No
*Type*: Array of [ParameterValue](aws-properties-datapipeline-pipeline-parametervalue.md)
*Update requires*: [No interruption](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-no-interrupt)
`PipelineObjects`
The objects that define the pipeline. These objects overwrite the existing pipeline definition. Not all objects, fields, and values can be updated. For information about restrictions, see [Editing Your Pipeline](https://docs.amazonaws.cn/datapipeline/latest/DeveloperGuide/dp-manage-pipeline-modify-console.html) in the *Amazon Data Pipeline Developer Guide*.
*Required*: No
*Type*: Array of [PipelineObject](aws-properties-datapipeline-pipeline-pipelineobject.md)
*Update requires*: [No interruption](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-no-interrupt)
`PipelineTags`
A list of arbitrary tags (key-value pairs) to associate with the pipeline, which you can use to control permissions. For more information, see [Controlling Access to Pipelines and Resources](https://docs.amazonaws.cn/datapipeline/latest/DeveloperGuide/dp-control-access.html) in the *Amazon Data Pipeline Developer Guide*.
*Required*: No
*Type*: Array of [PipelineTag](aws-properties-datapipeline-pipeline-pipelinetag.md)
*Update requires*: [No interruption](https://docs.amazonaws.cn/AWSCloudFormation/latest/UserGuide/using-cfn-updating-stacks-update-behaviors.html#update-no-interrupt)
## Return values
### Ref
When you pass the logical ID of this resource to the intrinsic `Ref` function, `Ref` returns the pipeline ID.
For more information about using the `Ref` function, see [https://docs.amazonaws.cn/AWSCloudFormation/latest/TemplateReference/intrinsic-function-reference-ref.html](https://docs.amazonaws.cn/AWSCloudFormation/latest/TemplateReference/intrinsic-function-reference-ref.html).
### Fn::GetAtt
####
`PipelineId`
The ID of the pipeline.
## Examples
The following data pipeline backs up data from an Amazon DynamoDB table to an Amazon S3 bucket. The pipeline uses the `HiveCopyActivity` activity to copy the data, and runs it once a day. The [roles](https://docs.amazonaws.cn/datapipeline/latest/DeveloperGuide/dp-iam-roles.html) for the pipeline and the pipeline resource are declared elsewhere in the same template.
**Topics**
+ [](#aws-resource-datapipeline-pipeline--examples--)
+ [](#aws-resource-datapipeline-pipeline--examples--)
###
#### JSON
```
"DynamoDBInputS3OutputHive": {
"Type": "AWS::DataPipeline::Pipeline",
"Properties": {
"Name": "DynamoDBInputS3OutputHive",
"Description": "Pipeline to backup DynamoDB data to S3",
"Activate": "true",
"ParameterObjects": [
{
"Id": "myDDBReadThroughputRatio",
"Attributes": [
{
"Key": "description",
"StringValue": "DynamoDB read throughput ratio"
},
{
"Key": "type",
"StringValue": "Double"
},
{
"Key": "default",
"StringValue": "0.2"
}
]
},
{
"Id": "myOutputS3Loc",
"Attributes": [
{
"Key": "description",
"StringValue": "S3 output bucket"
},
{
"Key": "type",
"StringValue": "AWS::S3::ObjectKey"
},
{
"Key": "default",
"StringValue": { "Fn::Join" : [ "", [ "s3://", { "Ref": "S3OutputLoc" } ] ] }
}
]
},
{
"Id": "myDDBTableName",
"Attributes": [
{
"Key": "description",
"StringValue": "DynamoDB Table Name "
},
{
"Key": "type",
"StringValue": "String"
}
]
}
],
"ParameterValues": [
{
"Id": "myDDBTableName",
"StringValue": { "Ref": "TableName" }
}
],
"PipelineObjects": [
{
"Id": "S3BackupLocation",
"Name": "Copy data to this S3 location",
"Fields": [
{
"Key": "type",
"StringValue": "S3DataNode"
},
{
"Key": "dataFormat",
"RefValue": "DDBExportFormat"
},
{
"Key": "directoryPath",
"StringValue": "#{myOutputS3Loc}/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}"
}
]
},
{
"Id": "DDBSourceTable",
"Name": "DDBSourceTable",
"Fields": [
{
"Key": "tableName",
"StringValue": "#{myDDBTableName}"
},
{
"Key": "type",
"StringValue": "DynamoDBDataNode"
},
{
"Key": "dataFormat",
"RefValue": "DDBExportFormat"
},
{
"Key": "readThroughputPercent",
"StringValue": "#{myDDBReadThroughputRatio}"
}
]
},
{
"Id": "DDBExportFormat",
"Name": "DDBExportFormat",
"Fields": [
{
"Key": "type",
"StringValue": "DynamoDBExportDataFormat"
}
]
},
{
"Id": "TableBackupActivity",
"Name": "TableBackupActivity",
"Fields": [
{
"Key": "resizeClusterBeforeRunning",
"StringValue": "true"
},
{
"Key": "type",
"StringValue": "HiveCopyActivity"
},
{
"Key": "input",
"RefValue": "DDBSourceTable"
},
{
"Key": "runsOn",
"RefValue": "EmrClusterForBackup"
},
{
"Key": "output",
"RefValue": "S3BackupLocation"
}
]
},
{
"Id": "DefaultSchedule",
"Name": "RunOnce",
"Fields": [
{
"Key": "occurrences",
"StringValue": "1"
},
{
"Key": "startAt",
"StringValue": "FIRST_ACTIVATION_DATE_TIME"
},
{
"Key": "type",
"StringValue": "Schedule"
},
{
"Key": "period",
"StringValue": "1 Day"
}
]
},
{
"Id": "Default",
"Name": "Default",
"Fields": [
{
"Key": "type",
"StringValue": "Default"
},
{
"Key": "scheduleType",
"StringValue": "cron"
},
{
"Key": "failureAndRerunMode",
"StringValue": "CASCADE"
},
{
"Key": "role",
"StringValue": "DataPipelineDefaultRole"
},
{
"Key": "resourceRole",
"StringValue": "DataPipelineDefaultResourceRole"
},
{
"Key": "schedule",
"RefValue": "DefaultSchedule"
}
]
},
{
"Id": "EmrClusterForBackup",
"Name": "EmrClusterForBackup",
"Fields": [
{
"Key": "terminateAfter",
"StringValue": "2 Hours"
},
{
"Key": "amiVersion",
"StringValue": "3.3.2"
},
{
"Key": "masterInstanceType",
"StringValue": "m1.medium"
},
{
"Key": "coreInstanceType",
"StringValue": "m1.medium"
},
{
"Key": "coreInstanceCount",
"StringValue": "1"
},
{
"Key": "type",
"StringValue": "EmrCluster"
}
]
}
]
}
}
```
###
#### YAML
```
DynamoDBInputS3OutputHive:
Type: AWS::DataPipeline::Pipeline
Properties:
Name: DynamoDBInputS3OutputHive
Description: "Pipeline to backup DynamoDB data to S3"
Activate: true
ParameterObjects:
-
Id: "myDDBReadThroughputRatio"
Attributes:
-
Key: "description"
StringValue: "DynamoDB read throughput ratio"
-
Key: "type"
StringValue: "Double"
-
Key: "default"
StringValue: "0.2"
-
Id: "myOutputS3Loc"
Attributes:
-
Key: "description"
StringValue: "S3 output bucket"
-
Key: "type"
StringValue: "AWS::S3::ObjectKey"
-
Key: "default"
StringValue:
Fn::Join:
- ""
-
- "s3://"
-
Ref: "S3OutputLoc"
-
Id: "myDDBTableName"
Attributes:
-
Key: "description"
StringValue: "DynamoDB Table Name "
-
Key: "type"
StringValue: "String"
ParameterValues:
-
Id: "myDDBTableName"
StringValue:
Ref: "TableName"
PipelineObjects:
-
Id: "S3BackupLocation"
Name: "Copy data to this S3 location"
Fields:
-
Key: "type"
StringValue: "S3DataNode"
-
Key: "dataFormat"
RefValue: "DDBExportFormat"
-
Key: "directoryPath"
StringValue: "#{myOutputS3Loc}/#{format(@scheduledStartTime, 'YYYY-MM-dd-HH-mm-ss')}"
-
Id: "DDBSourceTable"
Name: "DDBSourceTable"
Fields:
-
Key: "tableName"
StringValue: "#{myDDBTableName}"
-
Key: "type"
StringValue: "DynamoDBDataNode"
-
Key: "dataFormat"
RefValue: "DDBExportFormat"
-
Key: "readThroughputPercent"
StringValue: "#{myDDBReadThroughputRatio}"
-
Id: "DDBExportFormat"
Name: "DDBExportFormat"
Fields:
-
Key: "type"
StringValue: "DynamoDBExportDataFormat"
-
Id: "TableBackupActivity"
Name: "TableBackupActivity"
Fields:
-
Key: "resizeClusterBeforeRunning"
StringValue: "true"
-
Key: "type"
StringValue: "HiveCopyActivity"
-
Key: "input"
RefValue: "DDBSourceTable"
-
Key: "runsOn"
RefValue: "EmrClusterForBackup"
-
Key: "output"
RefValue: "S3BackupLocation"
-
Id: "DefaultSchedule"
Name: "RunOnce"
Fields:
-
Key: "occurrences"
StringValue: "1"
-
Key: "startAt"
StringValue: "FIRST_ACTIVATION_DATE_TIME"
-
Key: "type"
StringValue: "Schedule"
-
Key: "period"
StringValue: "1 Day"
-
Id: "Default"
Name: "Default"
Fields:
-
Key: "type"
StringValue: "Default"
-
Key: "scheduleType"
StringValue: "cron"
-
Key: "failureAndRerunMode"
StringValue: "CASCADE"
-
Key: "role"
StringValue: "DataPipelineDefaultRole"
-
Key: "resourceRole"
StringValue: "DataPipelineDefaultResourceRole"
-
Key: "schedule"
RefValue: "DefaultSchedule"
-
Id: "EmrClusterForBackup"
Name: "EmrClusterForBackup"
Fields:
-
Key: "terminateAfter"
StringValue: "2 Hours"
-
Key: "amiVersion"
StringValue: "3.3.2"
-
Key: "masterInstanceType"
StringValue: "m1.medium"
-
Key: "coreInstanceType"
StringValue: "m1.medium"
-
Key: "coreInstanceCount"
StringValue: "1"
-
Key: "type"
StringValue: "EmrCluster"
```
## See also
+ [Pipeline Object Reference](https://docs.amazonaws.cn/datapipeline/latest/DeveloperGuide/dp-pipeline-objects.html) in the *Amazon Data Pipeline Developer Guide*.
+ [PutPipelineDefinition](https://docs.amazonaws.cn/datapipeline/latest/APIReference/API_PutPipelineDefinition.html) in the *Amazon Data Pipeline API Reference*.