Differences and considerations for Hive on Amazon EMR

To pass variable values into Hive steps using the console

1. Open the Amazon EMR console at https://console.amazonaws.cn/emr.

2. Choose Create cluster.

3. In the Steps section, for Add Step, choose Hive Program from the list and Configure and add.

4. In the Add Step dialog, specify the parameters using the following table as a guide, and then choose Add.

   Field	Action
   Script S3 location*	Specify the URI where your script resides in Amazon S3. The value must be in the form BucketName/path/ScriptName. For example: s3://elasticmapreduce/samples/hive-ads/libs/response-time-stats.q.
   Input S3 location	Optionally, specify the URI where your input files reside in Amazon S3. The value must be in the form BucketName/path/. If specified, this will be passed to the Hive script as a parameter named INPUT. For example: s3://elasticmapreduce/samples/hive-ads/tables/.
   Output S3 location	Optionally, specify the URI where you want the output stored in Amazon S3. The value must be in the form BucketName/path. If specified, this will be passed to the Hive script as a parameter named OUTPUT. For example: s3://mybucket/hive-ads/output/.
   Arguments	Optionally, enter a list of arguments (space-separated strings) to pass to Hive. If you defined a path variable in your Hive script named ${SAMPLE}, for example: CREATE EXTERNAL TABLE logs (requestBeginTime STRING, requestEndTime STRING, hostname STRING) PARTITIONED BY (dt STRING) \ ROW FORMAT serde 'com.amazon.elasticmapreduce.JsonSerde' WITH SERDEPROPERTIES ( 'paths'='requestBeginTime, requestEndTime, hostname' ) LOCATION '${SAMPLE}/tables/impressions'; To pass a value for the variable, type the following in the Arguments window: -d SAMPLE=s3://elasticmapreduce/samples/hive-ads/.
   Action on Failure	This determines what the cluster does in response to any errors. The possible values for this setting are: Terminate cluster: If the step fails, terminate the cluster. If the cluster has termination protection enabled AND keep alive enabled, it will not terminate. Cancel and wait: If the step fails, cancel the remaining steps. If the cluster has keep alive enabled, the cluster will not terminate. Continue: If the step fails, continue to the next step.

5. Select values as necessary and choose Create cluster.

To pass variable values into Hive steps using the Amazon CLI

To pass variable values into Hive steps using the Amazon CLI, use the --steps parameter and include an arguments list.

• Note

  Linux line continuation characters (\) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^).

  aws emr create-cluster --name "Test cluster" --release-label emr-7.0.0 \
  --applications Name=Hive Name=Pig --use-default-roles --ec2-attributes KeyName=myKey --instance-type m5.xlarge --instance-count 3 \
  --steps Type=Hive,Name="Hive Program",ActionOnFailure=CONTINUE,Args=[-f,s3://elasticmapreduce/samples/hive-ads/libs/response-time-stats.q,-d,INPUT=s3://elasticmapreduce/samples/hive-ads/tables,-d,OUTPUT=s3://mybucket/hive-ads/output/,-d,SAMPLE=s3://elasticmapreduce/samples/hive-ads/]

  For more information on using Amazon EMR commands in the Amazon CLI, see https://docs.amazonaws.cn/cli/latest/reference/emr.

To pass variable values into Hive steps using the Java SDK

• The following example demonstrates how to pass variables into steps using the SDK. For more information, see Class StepFactory in the Amazon SDK for Java API Reference.

  StepFactory stepFactory = new StepFactory();
  
     StepConfig runHive = new StepConfig()
       .withName("Run Hive Script")
       .withActionOnFailure("TERMINATE_JOB_FLOW")
       .withHadoopJarStep(stepFactory.newRunHiveScriptStep(“s3://mybucket/script.q”,
        Lists.newArrayList(“-d”,”LIB= s3://elasticmapreduce/samples/hive-ads/lib”));