Class SparkSubmitJobDriver
The information about job driver for Spark submit.
Inheritance
Implements
Namespace: Amazon.CDK.AWS.StepFunctions.Tasks
Assembly: Amazon.CDK.Lib.dll
Syntax (csharp)
public class SparkSubmitJobDriver : Object, ISparkSubmitJobDriver
Syntax (vb)
Public Class SparkSubmitJobDriver
Inherits Object
Implements ISparkSubmitJobDriver
Remarks
ExampleMetadata: infused
Examples
new EmrContainersStartJobRun(this, "EMR Containers Start Job Run", new EmrContainersStartJobRunProps {
VirtualCluster = VirtualClusterInput.FromVirtualClusterId("de92jdei2910fwedz"),
ReleaseLabel = ReleaseLabel.EMR_6_2_0,
JobName = "EMR-Containers-Job",
JobDriver = new JobDriver {
SparkSubmitJobDriver = new SparkSubmitJobDriver {
EntryPoint = TaskInput.FromText("local:///usr/lib/spark/examples/src/main/python/pi.py")
}
},
ApplicationConfig = new [] { new ApplicationConfiguration {
Classification = Classification.SPARK_DEFAULTS,
Properties = new Dictionary<string, string> {
{ "spark.executor.instances", "1" },
{ "spark.executor.memory", "512M" }
}
} }
});
Synopsis
Constructors
SparkSubmitJobDriver() |
Properties
EntryPoint | The entry point of job application. |
EntryPointArguments | The arguments for a job application in a task input object containing an array of strings. |
SparkSubmitParameters | The Spark submit parameters that are used for job runs. |
Constructors
SparkSubmitJobDriver()
public SparkSubmitJobDriver()
Properties
EntryPoint
The entry point of job application.
public TaskInput EntryPoint { get; set; }
Property Value
Remarks
Length Constraints: Minimum length of 1. Maximum length of 256.
EntryPointArguments
The arguments for a job application in a task input object containing an array of strings.
public TaskInput EntryPointArguments { get; set; }
Property Value
Remarks
Length Constraints: Minimum length of 1. Maximum length of 10280.
Default: - No arguments defined
Type: sfn.TaskInput which expects payload as an array of strings
SparkSubmitParameters
The Spark submit parameters that are used for job runs.
public string SparkSubmitParameters { get; set; }
Property Value
System.String
Remarks
Length Constraints: Minimum length of 1. Maximum length of 102400.
Default: - No spark submit parameters