Class that makes it easy to define Hadoop Streaming steps.
CopyC#
See also: Hadoop Streaming
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey); AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials); HadoopJarStepConfig config = new StreamingStep { Inputs = new List<string> { "s3://elasticmapreduce/samples/wordcount/input" }, Output = "s3://my-bucket/output/", Mapper = "s3://elasticmapreduce/samples/wordcount/wordSplitter.py", Reducer = "aggregate" }.ToHadoopJarStepConfig(); StepConfig wordCount = new StepConfig { Name = "Word Count", ActionOnFailure = "TERMINATE_JOB_FLOW", HadoopJarStep = config }; RunJobFlowRequest request = new RunJobFlowRequest { Name = "Word Count", Steps = new List<StepConfig> { wordCount }, LogUri = "s3://log-bucket/", Instances = new JobFlowInstancesConfig { Ec2KeyName = "keypair", HadoopVersion = "0.20", InstanceCount = 5, KeepJobFlowAliveWhenNoSteps = true, MasterInstanceType = "m1.small", SlaveInstanceType = "m1.small" } }; RunJobFlowResult result = emr.RunJobFlow(request).RunJobFlowResult;
| C# |
public class StreamingStep
| All Members | Constructors | Methods | Properties | ||
| Icon | Member | Description |
|---|---|---|
| StreamingStep()()()() |
Creates a new default StreamingStep.
| |
| Equals(Object) | (Inherited from Object.) | |
| GetHashCode()()()() | Serves as a hash function for a particular type. (Inherited from Object.) | |
| GetType()()()() | Gets the type of the current instance. (Inherited from Object.) | |
| HadoopConfig |
Gets and sets the hadoopConfig.
| |
| Inputs |
Gets and sets list of step input paths.
| |
| Mapper |
Gets and sets the mapper.
| |
| Output |
Gets and sets the output path.
| |
| Reducer |
Gets and sets the reducer.
| |
| ToHadoopJarStepConfig()()()() |
Creates the final HadoopJarStepConfig once you are done configuring the step. You can use
this as you would any other HadoopJarStepConfig.
| |
| ToString()()()() | Returns a string that represents the current object. (Inherited from Object.) | |
| WithHadoopConfig(String, String) | Obsolete.
Add a Hadoop config override (-D value)
| |
| WithInputs(array<String>[]()[][]) | Obsolete.
Add more input paths to this step.
| |
| WithMapper(String) | Obsolete.
Set the mapper.
| |
| WithOutput(String) | Obsolete.
Set the output path for this step.
| |
| WithReducer(String) | Obsolete.
Set the reducer.
|
| Object | |
| StreamingStep | |