Transparent encryption in HDFS on Amazon EMR

To create encryption zones and their keys at cluster creation using the CLI

The hdfs-encryption-zones classification in the configuration API operation allows you to specify a key name and an encryption zone when you create a cluster. Amazon EMR creates this key in Hadoop KMS on your cluster and configures the encryption zone.

• Create a cluster with the following command.

  aws emr create-cluster --release-label emr-7.5.0 --instance-type m5.xlarge --instance-count 2 \
  --applications Name=App1 Name=App2 --configurations https://s3.amazonaws.com/amzn-s3-demo-bucket/myfolder/myConfig.json

  Note

  Linux line continuation characters (\) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^).

  myConfig.json:

  [
    {
      "Classification": "hdfs-encryption-zones",
      "Properties": {
        "/myHDFSPath1": "path1_key",
        "/myHDFSPath2": "path2_key"
      }
    }
  ]

To create encryption zones and their keys manually on the master node

1. Launch your cluster using an Amazon EMR release greater than 4.1.0.

2. Connect to the master node of the cluster with SSH.

3. Create a key within Hadoop KMS.

   $ hadoop key create path2_key
   path2_key has been successfully created with options Options{cipher='AES/CTR/NoPadding', bitLength=256, description='null', attributes=null}.
   KMSClientProvider[http://ip-x-x-x-x.ec2.internal:16000/kms/v1/] has been updated.

   Important

   Hadoop KMS requires your key names to be lowercase. If you use a key that has uppercase characters, then your cluster will fail during launch.

4. Create the encryption zone path in HDFS.

   $ hadoop fs -mkdir /myHDFSPath2

5. Make the HDFS path an encryption zone using the key that you created.

   $ hdfs crypto -createZone -keyName path2_key -path /myHDFSPath2
   Added encryption zone /myHDFSPath2

To create encryption zones and their keys manually using the Amazon CLI

• Add steps to create the KMS keys and encryption zones manually with the following command.

  aws emr add-steps --cluster-id j-2AXXXXXXGAPLF --steps Type=CUSTOM_JAR,Name="Create First Hadoop KMS Key",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=[/bin/bash,-c,"\"hadoop key create path1_key\""] \
  Type=CUSTOM_JAR,Name="Create First Hadoop HDFS Path",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=[/bin/bash,-c,"\"hadoop fs -mkdir /myHDFSPath1\""] \
  Type=CUSTOM_JAR,Name="Create First Encryption Zone",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=[/bin/bash,-c,"\"hdfs crypto -createZone -keyName path1_key -path /myHDFSPath1\""] \
  Type=CUSTOM_JAR,Name="Create Second Hadoop KMS Key",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=[/bin/bash,-c,"\"hadoop key create path2_key\""] \
  Type=CUSTOM_JAR,Name="Create Second Hadoop HDFS Path",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=[/bin/bash,-c,"\"hadoop fs -mkdir /myHDFSPath2\""] \
  Type=CUSTOM_JAR,Name="Create Second Encryption Zone",Jar="command-runner.jar",ActionOnFailure=CONTINUE,Args=[/bin/bash,-c,"\"hdfs crypto -createZone -keyName path2_key -path /myHDFSPath2\""] 
  

  Note

  Linux line continuation characters (\) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^).

To set Hadoop KMS ACLs using the CLI

• Create a cluster with Hadoop KMS with ACLs using the following command:

  aws emr create-cluster --release-label emr-7.5.0 --instance-type m5.xlarge --instance-count 2 \
  --applications Name=App1 Name=App2 --configurations https://s3.amazonaws.com/amzn-s3-demo-bucket/myfolder/myConfig.json

  Note

  Linux line continuation characters (\) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^).

  myConfig.json:

  [
      {
        "Classification": "hadoop-kms-acls",
        "Properties": {
          "hadoop.kms.blacklist.CREATE": "hdfs,foo,myBannedUser",
          "hadoop.kms.acl.ROLLOVER": "myAllowedUser"       
        }
      }
    ]
  

To disable Hadoop KMS cache using the CLI

• Create a cluster with Hadoop KMS hadoop.kms.cache.enable set to false, using the following command:

  aws emr create-cluster --release-label emr-7.5.0 --instance-type m5.xlarge --instance-count 2 \
  --applications Name=App1 Name=App2 --configurations https://s3.amazonaws.com/amzn-s3-demo-bucket/myfolder/myConfig.json

  Note

  Linux line continuation characters (\) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^).

  myConfig.json:

  [
      {
        "Classification": "hadoop-kms-site",
        "Properties": {
          "hadoop.kms.cache.enable": "false"
        }
      }
    ]
  

To set environment variables in the kms-env.sh script using the CLI

• Change settings in kms-env.sh via the hadoop-kms-env configuration. Create a cluster with Hadoop KMS using the following command:

  aws emr create-cluster --release-label emr-7.5.0 --instance-type m5.xlarge --instance-count 2 \
  --applications Name=App1 Name=App2 --configurations https://s3.amazonaws.com/amzn-s3-demo-bucket/myfolder/myConfig.json

  Note

  Linux line continuation characters (\) are included for readability. They can be removed or used in Linux commands. For Windows, remove them or replace with a caret (^).

  myConfig.json:

  [
    {
      "Classification": "hadoop-kms-env",
      "Properties": {     
      },
      "Configurations": [
        {
          "Classification": "export",
          "Properties": {
            "JAVA_LIBRARY_PATH": "/path/to/files",
            "KMS_SSL_KEYSTORE_FILE": "/non/Default/Path/.keystore",
            "KMS_SSL_KEYSTORE_PASS": "myPass"
          },
          "Configurations": [        
          ]
        }
      ]
    }
  ]