Interface S3DataSource.Builder
- All Superinterfaces:
Buildable
,CopyableBuilder<S3DataSource.Builder,
,S3DataSource> SdkBuilder<S3DataSource.Builder,
,S3DataSource> SdkPojo
- Enclosing class:
S3DataSource
-
Method Summary
Modifier and TypeMethodDescriptionattributeNames
(String... attributeNames) A list of one or more attribute names to use that are found in a specified augmented manifest file.attributeNames
(Collection<String> attributeNames) A list of one or more attribute names to use that are found in a specified augmented manifest file.instanceGroupNames
(String... instanceGroupNames) A list of names of instance groups that get data from the S3 data source.instanceGroupNames
(Collection<String> instanceGroupNames) A list of names of instance groups that get data from the S3 data source.s3DataDistributionType
(String s3DataDistributionType) If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specifyFullyReplicated
.s3DataDistributionType
(S3DataDistribution s3DataDistributionType) If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specifyFullyReplicated
.s3DataType
(String s3DataType) If you chooseS3Prefix
,S3Uri
identifies a key name prefix.s3DataType
(S3DataType s3DataType) If you chooseS3Prefix
,S3Uri
identifies a key name prefix.Depending on the value specified for theS3DataType
, identifies either a key name prefix or a manifest.Methods inherited from interface software.amazon.awssdk.utils.builder.CopyableBuilder
copy
Methods inherited from interface software.amazon.awssdk.utils.builder.SdkBuilder
applyMutation, build
Methods inherited from interface software.amazon.awssdk.core.SdkPojo
equalsBySdkFields, sdkFields
-
Method Details
-
s3DataType
If you choose
S3Prefix
,S3Uri
identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.If you choose
ManifestFile
,S3Uri
identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.If you choose
AugmentedManifestFile
, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training.AugmentedManifestFile
can only be used if the Channel's input mode isPipe
.- Parameters:
s3DataType
- If you chooseS3Prefix
,S3Uri
identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.If you choose
ManifestFile
,S3Uri
identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.If you choose
AugmentedManifestFile
, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training.AugmentedManifestFile
can only be used if the Channel's input mode isPipe
.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
s3DataType
If you choose
S3Prefix
,S3Uri
identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.If you choose
ManifestFile
,S3Uri
identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.If you choose
AugmentedManifestFile
, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training.AugmentedManifestFile
can only be used if the Channel's input mode isPipe
.- Parameters:
s3DataType
- If you chooseS3Prefix
,S3Uri
identifies a key name prefix. SageMaker uses all objects that match the specified key name prefix for model training.If you choose
ManifestFile
,S3Uri
identifies an object that is a manifest file containing a list of object keys that you want SageMaker to use for model training.If you choose
AugmentedManifestFile
, S3Uri identifies an object that is an augmented manifest file in JSON lines format. This file contains the data you want to use for model training.AugmentedManifestFile
can only be used if the Channel's input mode isPipe
.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
s3Uri
Depending on the value specified for the
S3DataType
, identifies either a key name prefix or a manifest. For example:-
A key name prefix might look like this:
s3://bucketname/exampleprefix/
-
A manifest might look like this:
s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of
S3Uri
. Note that the prefix must be a valid non-emptyS3Uri
that precludes users from specifying a manifest whose individualS3Uri
is sourced from different S3 buckets.The following code example shows a valid manifest format:
[ {"prefix": "s3://customer_bucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
]
This JSON is equivalent to the following
S3Uri
list:s3://customer_bucket/some/prefix/relative/path/to/custdata-1
s3://customer_bucket/some/prefix/relative/path/custdata-2
...
s3://customer_bucket/some/prefix/relative/path/custdata-N
The complete set of
S3Uri
in this manifest is the input data for the channel for this data source. The object that eachS3Uri
points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.
Your input bucket must be located in same Amazon Web Services region as your training job.
- Parameters:
s3Uri
- Depending on the value specified for theS3DataType
, identifies either a key name prefix or a manifest. For example:-
A key name prefix might look like this:
s3://bucketname/exampleprefix/
-
A manifest might look like this:
s3://bucketname/example.manifest
A manifest is an S3 object which is a JSON file consisting of an array of elements. The first element is a prefix which is followed by one or more suffixes. SageMaker appends the suffix elements to the prefix to get a full set of
S3Uri
. Note that the prefix must be a valid non-emptyS3Uri
that precludes users from specifying a manifest whose individualS3Uri
is sourced from different S3 buckets.The following code example shows a valid manifest format:
[ {"prefix": "s3://customer_bucket/some/prefix/"},
"relative/path/to/custdata-1",
"relative/path/custdata-2",
...
"relative/path/custdata-N"
]
This JSON is equivalent to the following
S3Uri
list:s3://customer_bucket/some/prefix/relative/path/to/custdata-1
s3://customer_bucket/some/prefix/relative/path/custdata-2
...
s3://customer_bucket/some/prefix/relative/path/custdata-N
The complete set of
S3Uri
in this manifest is the input data for the channel for this data source. The object that eachS3Uri
points to must be readable by the IAM role that SageMaker uses to perform tasks on your behalf.
Your input bucket must be located in same Amazon Web Services region as your training job.
-
- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
-
s3DataDistributionType
If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify
FullyReplicated
.If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify
ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (whenTrainingInputMode
is set toFile
), this copies 1/n of the number of objects.- Parameters:
s3DataDistributionType
- If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specifyFullyReplicated
.If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify
ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (whenTrainingInputMode
is set toFile
), this copies 1/n of the number of objects.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
s3DataDistributionType
If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specify
FullyReplicated
.If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify
ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (whenTrainingInputMode
is set toFile
), this copies 1/n of the number of objects.- Parameters:
s3DataDistributionType
- If you want SageMaker to replicate the entire dataset on each ML compute instance that is launched for model training, specifyFullyReplicated
.If you want SageMaker to replicate a subset of data on each ML compute instance that is launched for model training, specify
ShardedByS3Key
. If there are n ML compute instances launched for a training job, each instance gets approximately 1/n of the number of S3 objects. In this case, model training on each machine uses only the subset of training data.Don't choose more ML compute instances for training than available S3 objects. If you do, some nodes won't get any data and you will pay for nodes that aren't getting any training data. This applies in both File and Pipe modes. Keep this in mind when developing algorithms.
In distributed training, where you use multiple ML compute EC2 instances, you might choose
ShardedByS3Key
. If the algorithm requires copying training data to the ML storage volume (whenTrainingInputMode
is set toFile
), this copies 1/n of the number of objects.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
attributeNames
A list of one or more attribute names to use that are found in a specified augmented manifest file.
- Parameters:
attributeNames
- A list of one or more attribute names to use that are found in a specified augmented manifest file.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
attributeNames
A list of one or more attribute names to use that are found in a specified augmented manifest file.
- Parameters:
attributeNames
- A list of one or more attribute names to use that are found in a specified augmented manifest file.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
instanceGroupNames
A list of names of instance groups that get data from the S3 data source.
- Parameters:
instanceGroupNames
- A list of names of instance groups that get data from the S3 data source.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
instanceGroupNames
A list of names of instance groups that get data from the S3 data source.
- Parameters:
instanceGroupNames
- A list of names of instance groups that get data from the S3 data source.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-