Authentication with mTLS for Redshift streaming ingestion from Apache Kafka sources
Mutual transport-layer security (mTLS) provides the means for a server to authenticate a client it's sending information to, and for the client to authenticate the server. The benefit of using mTLS is to provide trusted authentication for a variety of use cases in several industry-vertical applications. These include use cases in financial, retail, government and healthcare industries. In the case of streaming ingestion to Redshift, authentication occurs between a server, which can be Amazon MSK, Apache Kafka, or Confluent Cloud, and an Amazon Redshift provisioned cluster or an Amazon Redshift Serverless workgroup.
This topic provides procedures and SQL-command examples that show ways to create an external schema that uses mTLS to authenticate between the Redshift client and any Apache Kafka server. The steps in this topic complement the full set of steps for setting up streaming ingestion from Apache Kafka sources. For more information, see Getting started with streaming ingestion from Apache Kafka sources.
Prerequisites for using mTLS for streaming ingestion
This section provides prerequisite steps for using mTLS for streaming ingestion with either Amazon Certificate Manager or Amazon Secrets Manager.
As a preliminary step, you must have or create a private certificate authority (PCA), which you can use to issue certificates that, among other functions, enable secure communication through secure communication channels. Amazon Private Certificate Authority (Private CA) is an available service that performs this function. For more information, see Creating a private CA in the Amazon Private Certificate Authority User Guide. After creating the Private CA, export the root CA certificate and save it to a file with a .pem extension.
To create a cluster that uses the CA certificate, do the following:
Using mTLS for streaming ingestion with Amazon Certificate Manager
The following procedure shows how to configure mTLS for Redshift streaming ingestion by using Amazon Certificate Manager (ACM) for certificate storage and management:
Request a private certificate through ACM. When you do this, select the PCA you created in the Prerequisites section as the certificate authority. ACM stores the signed certificate and attached private key for secure communication. For more information about managing certificates with ACM, see Issuing and managing certificates in the Amazon Certificate Manager User Guide.
For the IAM role that you use to manage your Redshift cluster or Amazon Redshift Serverless workgroup, attach the permission to export the certificate, which is acm:ExportCertificate. For more information about setting up the necessary IAM resources for streaming ingestion, see Setting up streaming ingestion from Kafka. Specify the same IAM role in the next step to create the external schema.
Note
Requests to Amazon Certificate Manager require an Internet gateway (IGW) or a NAT gateway (NGW) in your VPC. If your VPC doesn't have either an IGW or a NGW, do the following:
Use Secrets Manager instead of ACM to store your certificates.
Attach a Secrets Manager VPC endpoint to your VPC.
For information about using Secrets Manager with mTLS for streaming ingestion, see Using mTLS for streaming ingestion with Amazon Secrets Manager following.
Get the bootstrap broker URI for the Amazon MSK, Apache Kafka, or Confluent Cloud cluster. For information about getting the bootstrap broker URI for Amazon MSK, see Getting the bootstrap brokers for an Amazon MSK cluster in the Amazon Managed Streaming for Apache Kafka Developer Guide.
Run a SQL command such as the following example to create an external schema that maps the cluster to a Redshift external schema, using
mtls.Important parameters:
IAM_ROLE – The IAM role associated with the cluster, for streaming ingestion.
URI – The bootstrap broker URI for the cluster. Note that for Amazon MSK, port 9094 is specified for communicating with brokers for TLS encryption.
AUTHENTICATION_ARN – The ARN of the ACM certificate. The ARN is available in the ACM console when you choose the issued certificate.
After performing these configuration steps, you can create a Redshift materialized view that references the schema defined in the sample and then use REFRESH MATERIALIZED VIEW to stream data. For more information, see Getting started with streaming ingestion from Apache Kafka sources.
Using mTLS for streaming ingestion with Amazon Secrets Manager
You can configure mTLS for Redshift streaming ingestion by using Amazon Secrets Manager for certificate management if you don't want to reference the certificate in Amazon Certificate Manager. The following steps describe how to configure mTLS using Secrets Manager.
Create a certificate signing request and private key with your tool of choice. Then you can use the signing request to generate a signed certificate, using the same Amazon Private CA (PCA) that you used to generate the certificate for the cluster. For more information about issuing a certificate, see IssueCertificate in the Amazon Private Certificate Authority API Reference.
Extract the certificate using Amazon Private Certificate Authority. For more information, see Retrieve a private certificate in the Amazon Private Certificate Authority User Guide.
Store the certificate and private key generated in the previous step in Amazon Secrets Manager. Choose
Other type of secretand use the plaintext format. The key-value pairs should be in the format{"certificate":"<cert value>","privateKey":"<pkey value>"}, as in the following example. For more information about creating and managing secrets in Amazon Secrets Manager, see Create and manage secrets with Amazon Secrets Manager in the Amazon Secrets Manager User Guide.{"certificate":"-----BEGIN CERTIFICATE----- klhdslkfjahksgdfkgioeuyihbflahabhbdslv6akybeoiwv1hoaiusdhbahsbdi H4hAX8/eE96qCcjkpfT84EdvHzp6fC+/WwM0oXlwUEWlvfMCXNaG5D8SqRq3qA== -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- klhdslkfjahksgdfkgioeuyihbflahabhbdslv6akybeoiwv1hoaiusdhbahsbdi wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY -----END CERTIFICATE-----", "privateKey":"-----BEGIN PRIVATE KEY----- klhdslkfjahksgdfkgioeuyihbflahabhbdslv6akybeoiwv1hoaiusdhbahsbdi 7OD4m1dBEs3Fj++hDMH9rYRp99RqtCOf0EWOUe139KOilOsW+cyhAoc9Ci2+Jo/k 17u2N1iGILMQEZuCRtnJOkFYkw== -----END PRIVATE KEY-----"}Attach the permission policy to retrieve the secret to the IAM role that you use to manage your Amazon Redshift cluster or Amazon Redshift Serverless workgroup. This permission is
secretsmanager:GetSecretValue. For more information, see Set up authentication. For more information about managing IAM policies, see Edit IAM policies. Specify the same IAM role in the next step to create the external schema.In Redshift, run the SQL command to create the external schema. You use the AUTHENTICATION type
mtls. You also specify the URI of the cluster and the secret ARN in Amazon Secrets Manager.CREATE EXTERNAL SCHEMA my_schema FROM KAFKA IAM_ROLE 'arn:aws:iam::012345678901:role/my_role' AUTHENTICATION mtls URI 'b-1.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9094,b-2.myTestCluster.123z8u.c2.kafka.us-west-1.amazonaws.com:9094' SECRET_ARN 'arn:aws:secretsmanager:us-east-1:012345678910:secret:myMTLSSecret';
Important parameters:
IAM_ROLE – The IAM role associated with the cluster, for streaming ingestion.
URI – The bootstrap broker URI for the cluster. Note that for Amazon MSK, port 9094 is specified for communicating with brokers for TLS encryption.
SECRET_ARN – The ARN of the secret from Secrets Manager, containing the certificate to use for mTLS.
Enabling mTLS authentication for an existing external schema
If you have an existing external schema that you use for streaming ingestion and you want to implement mutual TLS for authentication, you can run a command such as the following, which specifies mTLS authentication and the ACM certificate ARN in ACM.
ALTER EXTERNAL SCHEMA schema_name AUTHENTICATION mtls AUTHENTICATION_ARN 'arn:aws:acm:Region:444455556666:certificate/certificate_ID';
Or you can specify mTLS authentication, with reference to the secret ARN in Amazon Secrets Manager.
ALTER EXTERNAL SCHEMA schema_name AUTHENTICATION mtls SECRET_ARN 'arn:aws:secretsmanager:us-east-1:012345678910:secret:myMTLSSecret';