Create a cluster with instance fleets or uniform instance groups
When you create a cluster and specify the configuration of the primary node, core nodes, and task nodes, you have two configuration options. You can use instance fleets or uniform instance groups. The configuration option you choose applies to all nodes, it applies for the lifetime of the cluster, and instance fleets and instance groups cannot coexist in a cluster. The instance fleets configuration is available in Amazon EMR version 4.8.0 and later, excluding 5.0.x versions.
You can use the Amazon EMR console, the Amazon CLI, or the Amazon EMR API to create clusters with
either configuration. When you use the create-cluster command from the
Amazon CLI, you use either the --instance-fleets parameters to create the
cluster using instance fleets or, alternatively, you use the
--instance-groups parameters to create it using uniform instance
groups.
The same is true using the Amazon EMR API. You use either the
InstanceGroups configuration to specify an array of
InstanceGroupConfig objects, or you use the
InstanceFleets configuration to specify an array of
InstanceFleetConfig objects.
In the new Amazon EMR console, you can choose to use either instance groups or instance fleets when you create a cluster, and you have the option to use Spot Instances with each. With the old Amazon EMR console, if you use the default Quick Options settings when you create your cluster, Amazon EMR applies the uniform instance groups configuration to the cluster and uses On-Demand Instances. To use Spot Instances with uniform instance groups, or to configure instance fleets and other customizations, choose Advanced Options.
Instance fleets
The instance fleets configuration offers the widest variety of provisioning options for Amazon EC2 instances. Each node type has a single instance fleet, and using a task instance fleet is optional. You can specify up to five EC2 instance types per fleet, or 30 EC2 instance types per fleet when you create a cluster using the Amazon CLI or Amazon EMR API and an allocation strategy for On-Demand and Spot Instances. For the core and task instance fleets, you assign a target capacity for On-Demand Instances, and another for Spot Instances. Amazon EMR chooses any mix of the specified instance types to fulfill the target capacities, provisioning both On-Demand and Spot Instances.
For the primary node type, Amazon EMR chooses a single instance type from your list of instances, and you specify whether it's provisioned as an On-Demand or Spot Instance. Instance fleets also provide additional options for Spot Instance and On-Demand purchases. Spot Instance options include a timeout that specifies an action to take if Spot capacity can't be provisioned, and a preferred allocation strategy (capacity-optimized) for launching Spot Instance fleets. On-Demand Instance fleets can also be launched using the allocation strategy (lowest-price) option. If you use a service role that is not the EMR default service role, or use an EMR managed policy in your service role, you need to add additional permissions to the custom cluster service role to enable the allocation strategy option. For more information, see Service role for Amazon EMR (EMR role).
For more information about configuring instance fleets, see Configure instance fleets.
Uniform instance groups
Uniform instance groups offer a simpler setup than instance fleets. Each Amazon EMR cluster can include up to 50 instance groups: one primary instance group that contains one Amazon EC2 instance, a core instance group that contains one or more EC2 instances, and up to 48 optional task instance groups. Each core and task instance group can contain any number of Amazon EC2 instances. You can scale each instance group by adding and removing Amazon EC2 instances manually, or you can set up automatic scaling. For information about adding and removing instances, see Use cluster scaling.
For more information about configuring uniform instance groups, see Configure uniform instance groups.