Common concepts for Amazon EMR API calls
When you write an application that calls the Amazon EMR API, there are several concepts that apply when calling one of the wrapper functions of an SDK.
Topics
Endpoints for Amazon EMR
An endpoint is a URL that is the entry point for a web service. Every web service request
must contain an endpoint. The endpoint specifies the Amazon Region where clusters are
created, described, or terminated. It has the form
elasticmapreduce.
.
If you specify the general endpoint (regionname
.amazonaws.com.cnelasticmapreduce.amazonaws.com.cn
), Amazon EMR
directs your request to an endpoint in the default Region. For accounts created on
or after March 8, 2013, the default Region is us-west-2; for older accounts, the
default Region is us-east-1.
For more information about the endpoints for Amazon EMR, see Regions and endpoints in the Amazon Web Services General Reference.
Specifying cluster parameters in Amazon EMR
The Instances
parameters enable you to configure the type and
number of EC2 instances to create nodes to process the data. Hadoop spreads the
processing of the data across multiple cluster nodes. The master node is responsible for
keeping track of the health of the core and task nodes and polling the nodes for job
result status. The core and task nodes do the actual processing of the data. If you have
a single-node cluster, the node serves as both the master and a core node.
The KeepJobAlive
parameter in a
RunJobFlow
request determines whether to terminate the cluster
when it runs out of cluster steps to execute. Set this value to False
when
you know that the cluster is running as expected. When you are troubleshooting the job
flow and adding steps while the cluster execution is suspended, set the value to
True
. This reduces the amount of time and expense of uploading the
results to Amazon Simple Storage Service (Amazon S3), only to repeat the process after modifying a step to restart the
cluster.
If KeepJobAlive
is true
, after successfully
getting the cluster to complete its work, you must send a
TerminateJobFlows
request or the cluster continues to run and
generate Amazon charges.
For more information about parameters that are unique to
RunJobFlow
, see RunJobFlow. For more information about the generic parameters in the
request, see Common request parameters.
Availability Zones in Amazon EMR
Amazon EMR uses EC2 instances as nodes to process clusters. These EC2 instances have locations composed of Availability Zones and Regions. Regions are dispersed and located in separate geographic areas. Availability Zones are distinct locations within a Region insulated from failures in other Availability Zones. Each Availability Zone provides inexpensive, low-latency network connectivity to other Availability Zones in the same Region. For a list of the Regions and endpoints for Amazon EMR, see Regions and endpoints in the Amazon Web Services General Reference.
The AvailabilityZone
parameter specifies the general location
of the cluster. This parameter is optional and, in general, we discourage its use. When
AvailabilityZone
is not specified Amazon EMR automatically picks the
best AvailabilityZone
value for the cluster. You might find this
parameter useful if you want to colocate your instances with other existing running
instances, and your cluster needs to read or write data from those instances. For more
information, see the Amazon EC2 User Guide.
How to use additional files and libraries in Amazon EMR clusters
There are times when you might like to use additional files or custom libraries with your mapper or reducer applications. For example, you might like to use a library that converts a PDF file into plain text.
To cache a file for the mapper or reducer to use when using Hadoop streaming
-
In the JAR
args
field:, add the following argument:-cacheFile s3://bucket/path_to_executable#local_path
The file,
local_path
, is in the working directory of the mapper, which could reference the file.