Interface S3CsvSource.Builder
- All Superinterfaces:
Buildable
,CopyableBuilder<S3CsvSource.Builder,
,S3CsvSource> SdkBuilder<S3CsvSource.Builder,
,S3CsvSource> SdkPojo
- Enclosing class:
S3CsvSource
-
Method Summary
Modifier and TypeMethodDescriptiondefault S3CsvSource.Builder
additionalOptions
(Consumer<S3DirectSourceAdditionalOptions.Builder> additionalOptions) Specifies additional connection options.additionalOptions
(S3DirectSourceAdditionalOptions additionalOptions) Specifies additional connection options.compressionType
(String compressionType) Specifies how the data is compressed.compressionType
(CompressionType compressionType) Specifies how the data is compressed.Specifies a character to use for escaping.exclusions
(String... exclusions) A string containing a JSON list of Unix-style glob patterns to exclude.exclusions
(Collection<String> exclusions) A string containing a JSON list of Unix-style glob patterns to exclude.groupFiles
(String groupFiles) Grouping files is turned on by default when the input contains more than 50,000 files.The target group size in bytes.This option controls the duration in milliseconds after which the s3 listing is likely to be consistent.maxFilesInBand
(Integer maxFilesInBand) This option specifies the maximum number of files to save from the last maxBand seconds.A Boolean value that specifies whether a single record can span multiple lines.The name of the data store.optimizePerformance
(Boolean optimizePerformance) A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats.outputSchemas
(Collection<GlueSchema> outputSchemas) Specifies the data schema for the S3 CSV source.outputSchemas
(Consumer<GlueSchema.Builder>... outputSchemas) Specifies the data schema for the S3 CSV source.outputSchemas
(GlueSchema... outputSchemas) Specifies the data schema for the S3 CSV source.A list of the Amazon S3 paths to read from.paths
(Collection<String> paths) A list of the Amazon S3 paths to read from.Specifies the character to use for quoting.Specifies the character to use for quoting.If set to true, recursively reads files in all subdirectories under the specified paths.Specifies the delimiter character.Specifies the delimiter character.A Boolean value that specifies whether to skip the first data line.withHeader
(Boolean withHeader) A Boolean value that specifies whether to treat the first line as a header.writeHeader
(Boolean writeHeader) A Boolean value that specifies whether to write the header to output.Methods inherited from interface software.amazon.awssdk.utils.builder.CopyableBuilder
copy
Methods inherited from interface software.amazon.awssdk.utils.builder.SdkBuilder
applyMutation, build
Methods inherited from interface software.amazon.awssdk.core.SdkPojo
equalsBySdkFields, sdkFields
-
Method Details
-
name
The name of the data store.
- Parameters:
name
- The name of the data store.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
paths
A list of the Amazon S3 paths to read from.
- Parameters:
paths
- A list of the Amazon S3 paths to read from.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
paths
A list of the Amazon S3 paths to read from.
- Parameters:
paths
- A list of the Amazon S3 paths to read from.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
compressionType
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are
"gzip"
and"bzip"
).- Parameters:
compressionType
- Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are"gzip"
and"bzip"
).- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
compressionType
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are
"gzip"
and"bzip"
).- Parameters:
compressionType
- Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are"gzip"
and"bzip"
).- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
exclusions
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.
- Parameters:
exclusions
- A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
exclusions
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.
- Parameters:
exclusions
- A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
groupSize
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files,
"groupFiles"
must be set to"inPartition"
for this to take effect.- Parameters:
groupSize
- The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files,"groupFiles"
must be set to"inPartition"
for this to take effect.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
groupFiles
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to
"none"
.- Parameters:
groupFiles
- Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to"none"
.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
recurse
If set to true, recursively reads files in all subdirectories under the specified paths.
- Parameters:
recurse
- If set to true, recursively reads files in all subdirectories under the specified paths.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
maxBand
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
- Parameters:
maxBand
- This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
maxFilesInBand
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
- Parameters:
maxFilesInBand
- This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
additionalOptions
Specifies additional connection options.
- Parameters:
additionalOptions
- Specifies additional connection options.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
additionalOptions
default S3CsvSource.Builder additionalOptions(Consumer<S3DirectSourceAdditionalOptions.Builder> additionalOptions) Specifies additional connection options.
This is a convenience method that creates an instance of theS3DirectSourceAdditionalOptions.Builder
avoiding the need to create one manually viaS3DirectSourceAdditionalOptions.builder()
.When the
Consumer
completes,SdkBuilder.build()
is called immediately and its result is passed toadditionalOptions(S3DirectSourceAdditionalOptions)
.- Parameters:
additionalOptions
- a consumer that will call methods onS3DirectSourceAdditionalOptions.Builder
- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
separator
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
- Parameters:
separator
- Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
separator
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
- Parameters:
separator
- Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
escaper
Specifies a character to use for escaping. This option is used only when reading CSV files. The default value is
none
. If enabled, the character which immediately follows is used as-is, except for a small set of well-known escapes (\n
,\r
,\t
, and\0
).- Parameters:
escaper
- Specifies a character to use for escaping. This option is used only when reading CSV files. The default value isnone
. If enabled, the character which immediately follows is used as-is, except for a small set of well-known escapes (\n
,\r
,\t
, and\0
).- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
quoteChar
Specifies the character to use for quoting. The default is a double quote:
'"'
. Set this to-1
to turn off quoting entirely.- Parameters:
quoteChar
- Specifies the character to use for quoting. The default is a double quote:'"'
. Set this to-1
to turn off quoting entirely.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
quoteChar
Specifies the character to use for quoting. The default is a double quote:
'"'
. Set this to-1
to turn off quoting entirely.- Parameters:
quoteChar
- Specifies the character to use for quoting. The default is a double quote:'"'
. Set this to-1
to turn off quoting entirely.- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-
multiline
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is
False
, which allows for more aggressive file-splitting during parsing.- Parameters:
multiline
- A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value isFalse
, which allows for more aggressive file-splitting during parsing.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
withHeader
A Boolean value that specifies whether to treat the first line as a header. The default value is
False
.- Parameters:
withHeader
- A Boolean value that specifies whether to treat the first line as a header. The default value isFalse
.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
writeHeader
A Boolean value that specifies whether to write the header to output. The default value is
True
.- Parameters:
writeHeader
- A Boolean value that specifies whether to write the header to output. The default value isTrue
.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
skipFirst
A Boolean value that specifies whether to skip the first data line. The default value is
False
.- Parameters:
skipFirst
- A Boolean value that specifies whether to skip the first data line. The default value isFalse
.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
optimizePerformance
A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.
- Parameters:
optimizePerformance
- A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
outputSchemas
Specifies the data schema for the S3 CSV source.
- Parameters:
outputSchemas
- Specifies the data schema for the S3 CSV source.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
outputSchemas
Specifies the data schema for the S3 CSV source.
- Parameters:
outputSchemas
- Specifies the data schema for the S3 CSV source.- Returns:
- Returns a reference to this object so that method calls can be chained together.
-
outputSchemas
Specifies the data schema for the S3 CSV source.
This is a convenience method that creates an instance of theGlueSchema.Builder
avoiding the need to create one manually viaGlueSchema.builder()
.When the
Consumer
completes,SdkBuilder.build()
is called immediately and its result is passed tooutputSchemas(List<GlueSchema>)
.- Parameters:
outputSchemas
- a consumer that will call methods onGlueSchema.Builder
- Returns:
- Returns a reference to this object so that method calls can be chained together.
- See Also:
-