Class S3CsvSource
- All Implemented Interfaces:
Serializable
,SdkPojo
,ToCopyableBuilder<S3CsvSource.Builder,
S3CsvSource>
Specifies a command-separated value (CSV) data store stored in Amazon S3.
- See Also:
-
Nested Class Summary
-
Method Summary
Modifier and TypeMethodDescriptionSpecifies additional connection options.static S3CsvSource.Builder
builder()
final CompressionType
Specifies how the data is compressed.final String
Specifies how the data is compressed.final boolean
final boolean
equalsBySdkFields
(Object obj) Indicates whether some other object is "equal to" this one by SDK fields.final String
escaper()
Specifies a character to use for escaping.A string containing a JSON list of Unix-style glob patterns to exclude.final <T> Optional
<T> getValueForField
(String fieldName, Class<T> clazz) final String
Grouping files is turned on by default when the input contains more than 50,000 files.final String
The target group size in bytes.final boolean
For responses, this returns true if the service returned a value for the Exclusions property.final int
hashCode()
final boolean
For responses, this returns true if the service returned a value for the OutputSchemas property.final boolean
hasPaths()
For responses, this returns true if the service returned a value for the Paths property.final Integer
maxBand()
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent.final Integer
This option specifies the maximum number of files to save from the last maxBand seconds.final Boolean
A Boolean value that specifies whether a single record can span multiple lines.final String
name()
The name of the data store.final Boolean
A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats.final List
<GlueSchema> Specifies the data schema for the S3 CSV source.paths()
A list of the Amazon S3 paths to read from.final QuoteChar
Specifies the character to use for quoting.final String
Specifies the character to use for quoting.final Boolean
recurse()
If set to true, recursively reads files in all subdirectories under the specified paths.final Separator
Specifies the delimiter character.final String
Specifies the delimiter character.static Class
<? extends S3CsvSource.Builder> final Boolean
A Boolean value that specifies whether to skip the first data line.Take this object and create a builder that contains all of the current property values of this object.final String
toString()
Returns a string representation of this object.final Boolean
A Boolean value that specifies whether to treat the first line as a header.final Boolean
A Boolean value that specifies whether to write the header to output.Methods inherited from interface software.amazon.awssdk.utils.builder.ToCopyableBuilder
copy
-
Method Details
-
name
The name of the data store.
- Returns:
- The name of the data store.
-
hasPaths
public final boolean hasPaths()For responses, this returns true if the service returned a value for the Paths property. This DOES NOT check that the value is non-empty (for which, you should check theisEmpty()
method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified. -
paths
A list of the Amazon S3 paths to read from.
Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.
This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the
hasPaths()
method.- Returns:
- A list of the Amazon S3 paths to read from.
-
compressionType
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are
"gzip"
and"bzip"
).If the service returns an enum value that is not available in the current SDK version,
compressionType
will returnCompressionType.UNKNOWN_TO_SDK_VERSION
. The raw value returned by the service is available fromcompressionTypeAsString()
.- Returns:
- Specifies how the data is compressed. This is generally not necessary if the data has a standard file
extension. Possible values are
"gzip"
and"bzip"
). - See Also:
-
compressionTypeAsString
Specifies how the data is compressed. This is generally not necessary if the data has a standard file extension. Possible values are
"gzip"
and"bzip"
).If the service returns an enum value that is not available in the current SDK version,
compressionType
will returnCompressionType.UNKNOWN_TO_SDK_VERSION
. The raw value returned by the service is available fromcompressionTypeAsString()
.- Returns:
- Specifies how the data is compressed. This is generally not necessary if the data has a standard file
extension. Possible values are
"gzip"
and"bzip"
). - See Also:
-
hasExclusions
public final boolean hasExclusions()For responses, this returns true if the service returned a value for the Exclusions property. This DOES NOT check that the value is non-empty (for which, you should check theisEmpty()
method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified. -
exclusions
A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.
Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.
This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the
hasExclusions()
method.- Returns:
- A string containing a JSON list of Unix-style glob patterns to exclude. For example, "[\"**.pdf\"]" excludes all PDF files.
-
groupSize
The target group size in bytes. The default is computed based on the input data size and the size of your cluster. When there are fewer than 50,000 input files,
"groupFiles"
must be set to"inPartition"
for this to take effect.- Returns:
- The target group size in bytes. The default is computed based on the input data size and the size of your
cluster. When there are fewer than 50,000 input files,
"groupFiles"
must be set to"inPartition"
for this to take effect.
-
groupFiles
Grouping files is turned on by default when the input contains more than 50,000 files. To turn on grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when there are more than 50,000 files, set this parameter to
"none"
.- Returns:
- Grouping files is turned on by default when the input contains more than 50,000 files. To turn on
grouping with fewer than 50,000 files, set this parameter to "inPartition". To disable grouping when
there are more than 50,000 files, set this parameter to
"none"
.
-
recurse
If set to true, recursively reads files in all subdirectories under the specified paths.
- Returns:
- If set to true, recursively reads files in all subdirectories under the specified paths.
-
maxBand
This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
- Returns:
- This option controls the duration in milliseconds after which the s3 listing is likely to be consistent. Files with modification timestamps falling within the last maxBand milliseconds are tracked specially when using JobBookmarks to account for Amazon S3 eventual consistency. Most users don't need to set this option. The default is 900000 milliseconds, or 15 minutes.
-
maxFilesInBand
This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
- Returns:
- This option specifies the maximum number of files to save from the last maxBand seconds. If this number is exceeded, extra files are skipped and only processed in the next job run.
-
additionalOptions
Specifies additional connection options.
- Returns:
- Specifies additional connection options.
-
separator
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
If the service returns an enum value that is not available in the current SDK version,
separator
will returnSeparator.UNKNOWN_TO_SDK_VERSION
. The raw value returned by the service is available fromseparatorAsString()
.- Returns:
- Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
- See Also:
-
separatorAsString
Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
If the service returns an enum value that is not available in the current SDK version,
separator
will returnSeparator.UNKNOWN_TO_SDK_VERSION
. The raw value returned by the service is available fromseparatorAsString()
.- Returns:
- Specifies the delimiter character. The default is a comma: ",", but any other character can be specified.
- See Also:
-
escaper
Specifies a character to use for escaping. This option is used only when reading CSV files. The default value is
none
. If enabled, the character which immediately follows is used as-is, except for a small set of well-known escapes (\n
,\r
,\t
, and\0
).- Returns:
- Specifies a character to use for escaping. This option is used only when reading CSV files. The default
value is
none
. If enabled, the character which immediately follows is used as-is, except for a small set of well-known escapes (\n
,\r
,\t
, and\0
).
-
quoteChar
Specifies the character to use for quoting. The default is a double quote:
'"'
. Set this to-1
to turn off quoting entirely.If the service returns an enum value that is not available in the current SDK version,
quoteChar
will returnQuoteChar.UNKNOWN_TO_SDK_VERSION
. The raw value returned by the service is available fromquoteCharAsString()
.- Returns:
- Specifies the character to use for quoting. The default is a double quote:
'"'
. Set this to-1
to turn off quoting entirely. - See Also:
-
quoteCharAsString
Specifies the character to use for quoting. The default is a double quote:
'"'
. Set this to-1
to turn off quoting entirely.If the service returns an enum value that is not available in the current SDK version,
quoteChar
will returnQuoteChar.UNKNOWN_TO_SDK_VERSION
. The raw value returned by the service is available fromquoteCharAsString()
.- Returns:
- Specifies the character to use for quoting. The default is a double quote:
'"'
. Set this to-1
to turn off quoting entirely. - See Also:
-
multiline
A Boolean value that specifies whether a single record can span multiple lines. This can occur when a field contains a quoted new-line character. You must set this option to True if any record spans multiple lines. The default value is
False
, which allows for more aggressive file-splitting during parsing.- Returns:
- A Boolean value that specifies whether a single record can span multiple lines. This can occur when a
field contains a quoted new-line character. You must set this option to True if any record spans multiple
lines. The default value is
False
, which allows for more aggressive file-splitting during parsing.
-
withHeader
A Boolean value that specifies whether to treat the first line as a header. The default value is
False
.- Returns:
- A Boolean value that specifies whether to treat the first line as a header. The default value is
False
.
-
writeHeader
A Boolean value that specifies whether to write the header to output. The default value is
True
.- Returns:
- A Boolean value that specifies whether to write the header to output. The default value is
True
.
-
skipFirst
A Boolean value that specifies whether to skip the first data line. The default value is
False
.- Returns:
- A Boolean value that specifies whether to skip the first data line. The default value is
False
.
-
optimizePerformance
A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.
- Returns:
- A Boolean value that specifies whether to use the advanced SIMD CSV reader along with Apache Arrow based columnar memory formats. Only available in Glue version 3.0.
-
hasOutputSchemas
public final boolean hasOutputSchemas()For responses, this returns true if the service returned a value for the OutputSchemas property. This DOES NOT check that the value is non-empty (for which, you should check theisEmpty()
method on the property). This is useful because the SDK will never return a null collection or map, but you may need to differentiate between the service returning nothing (or null) and the service returning an empty collection or map. For requests, this returns true if a value for the property was specified in the request builder, and false if a value was not specified. -
outputSchemas
Specifies the data schema for the S3 CSV source.
Attempts to modify the collection returned by this method will result in an UnsupportedOperationException.
This method will never return null. If you would like to know whether the service returned this field (so that you can differentiate between null and empty), you can use the
hasOutputSchemas()
method.- Returns:
- Specifies the data schema for the S3 CSV source.
-
toBuilder
Description copied from interface:ToCopyableBuilder
Take this object and create a builder that contains all of the current property values of this object.- Specified by:
toBuilder
in interfaceToCopyableBuilder<S3CsvSource.Builder,
S3CsvSource> - Returns:
- a builder for type T
-
builder
-
serializableBuilderClass
-
hashCode
public final int hashCode() -
equals
-
equalsBySdkFields
Description copied from interface:SdkPojo
Indicates whether some other object is "equal to" this one by SDK fields. An SDK field is a modeled, non-inherited field in anSdkPojo
class, and is generated based on a service model.If an
SdkPojo
class does not have any inherited fields,equalsBySdkFields
andequals
are essentially the same.- Specified by:
equalsBySdkFields
in interfaceSdkPojo
- Parameters:
obj
- the object to be compared with- Returns:
- true if the other object equals to this object by sdk fields, false otherwise.
-
toString
Returns a string representation of this object. This is useful for testing and debugging. Sensitive data will be redacted from this string using a placeholder value. -
getValueForField
-
sdkFields
-