Class CfnDeliveryStream.OrcSerDeProperty
A serializer to use for converting data to the ORC format before storing it in Amazon S3.
Inheritance
Implements
Namespace: Amazon.CDK.AWS.KinesisFirehose
Assembly: Amazon.CDK.AWS.KinesisFirehose.dll
Syntax (csharp)
public class OrcSerDeProperty : Object, CfnDeliveryStream.IOrcSerDeProperty
Syntax (vb)
Public Class OrcSerDeProperty
Inherits Object
Implements CfnDeliveryStream.IOrcSerDeProperty
Remarks
For more information, see Apache ORC .
ExampleMetadata: fixture=_generated
Examples
// The code below shows an example of how to instantiate this type.
// The values are placeholders you should change.
using Amazon.CDK.AWS.KinesisFirehose;
var orcSerDeProperty = new OrcSerDeProperty {
BlockSizeBytes = 123,
BloomFilterColumns = new [] { "bloomFilterColumns" },
BloomFilterFalsePositiveProbability = 123,
Compression = "compression",
DictionaryKeyThreshold = 123,
EnablePadding = false,
FormatVersion = "formatVersion",
PaddingTolerance = 123,
RowIndexStride = 123,
StripeSizeBytes = 123
};
Synopsis
Constructors
OrcSerDeProperty() |
Properties
BlockSizeBytes | The Hadoop Distributed File System (HDFS) block size. |
BloomFilterColumns | The column names for which you want Kinesis Data Firehose to create bloom filters. |
BloomFilterFalsePositiveProbability | The Bloom filter false positive probability (FPP). |
Compression | The compression code to use over data blocks. |
DictionaryKeyThreshold | Represents the fraction of the total number of non-null rows. |
EnablePadding | Set this to |
FormatVersion | The version of the file to write. |
PaddingTolerance | A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size. |
RowIndexStride | The number of rows between index entries. |
StripeSizeBytes | The number of bytes in each stripe. |
Constructors
OrcSerDeProperty()
public OrcSerDeProperty()
Properties
BlockSizeBytes
The Hadoop Distributed File System (HDFS) block size.
public Nullable<double> BlockSizeBytes { get; set; }
Property Value
System.Nullable<System.Double>
Remarks
This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is 256 MiB and the minimum is 64 MiB. Kinesis Data Firehose uses this value for padding calculations.
BloomFilterColumns
The column names for which you want Kinesis Data Firehose to create bloom filters.
public string[] BloomFilterColumns { get; set; }
Property Value
System.String[]
Remarks
BloomFilterFalsePositiveProbability
The Bloom filter false positive probability (FPP).
public Nullable<double> BloomFilterFalsePositiveProbability { get; set; }
Property Value
System.Nullable<System.Double>
Remarks
The lower the FPP, the bigger the Bloom filter. The default value is 0.05, the minimum is 0, and the maximum is 1.
Compression
The compression code to use over data blocks.
public string Compression { get; set; }
Property Value
System.String
Remarks
DictionaryKeyThreshold
Represents the fraction of the total number of non-null rows.
public Nullable<double> DictionaryKeyThreshold { get; set; }
Property Value
System.Nullable<System.Double>
Remarks
To turn off dictionary encoding, set this fraction to a number that is less than the number of distinct keys in a dictionary. To always use dictionary encoding, set this threshold to 1.
EnablePadding
Set this to true
to indicate that you want stripes to be padded to the HDFS block boundaries.
public object EnablePadding { get; set; }
Property Value
System.Object
Remarks
This is useful if you intend to copy the data from Amazon S3 to HDFS before querying. The default is false
.
FormatVersion
The version of the file to write.
public string FormatVersion { get; set; }
Property Value
System.String
Remarks
The possible values are V0_11
and V0_12
. The default is V0_12
.
PaddingTolerance
A number between 0 and 1 that defines the tolerance for block padding as a decimal fraction of stripe size.
public Nullable<double> PaddingTolerance { get; set; }
Property Value
System.Nullable<System.Double>
Remarks
The default value is 0.05, which means 5 percent of stripe size.
For the default values of 64 MiB ORC stripes and 256 MiB HDFS blocks, the default block padding tolerance of 5 percent reserves a maximum of 3.2 MiB for padding within the 256 MiB block. In such a case, if the available size within the block is more than 3.2 MiB, a new, smaller stripe is inserted to fit within that space. This ensures that no stripe crosses block boundaries and causes remote reads within a node-local task.
Kinesis Data Firehose ignores this parameter when EnablePadding
is false
.
RowIndexStride
The number of rows between index entries.
public Nullable<double> RowIndexStride { get; set; }
Property Value
System.Nullable<System.Double>
Remarks
The default is 10,000 and the minimum is 1,000.
StripeSizeBytes
The number of bytes in each stripe.
public Nullable<double> StripeSizeBytes { get; set; }
Property Value
System.Nullable<System.Double>
Remarks
The default is 64 MiB and the minimum is 8 MiB.