Creating a BigQuery connection

To configure a connection to BigQuery:

1. In Google Cloud Platform, create and identify relevant resources:

   • Create or identify a GCP project containing BigQuery tables you would like to connect to.

   • Enable the BigQuery API. For more information, see Use the BigQuery Storage Read API to read table data .

2. In Google Cloud Platform, create and export service account credentials:

   You can use the BigQuery credentials wizard to expedite this step: Create credentials.

   To create a service account in GCP, follow the tutorial available in Create service accounts.

   • When selecting project, select the project containing your BigQuery table.

   • When selecting GCP IAM roles for your service account, add or create a role that would grant appropriate permissions to run BigQuery jobs to read, write or create BigQuery tables.

   To create credentials for your service account, follow the tutorial available in Create a service account key.

   • When selecting key type, select JSON.

   You should now have downloaded a JSON file with credentials for your service account. It should look similar to the following:

   {
     "type": "service_account",
     "project_id": "*****",
     "private_key_id": "*****",
     "private_key": "*****",
     "client_email": "*****",
     "client_id": "*****",
     "auth_uri": "https://accounts.google.com/o/oauth2/auth",
     "token_uri": "https://oauth2.googleapis.com/token",
     "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
     "client_x509_cert_url": "*****",
     "universe_domain": "googleapis.com"
   }

3. base64 encode your downloaded credentials file. On an Amazon CloudShell session or similar, you can do this from the command line by running cat credentialsFile.json | base64 -w 0. Retain the output of this command, credentialString.

4. In Amazon Secrets Manager, create a secret using your Google Cloud Platform credentials. To create a secret in Secrets Manager, follow the tutorial available in Create an Amazon Secrets Manager secret in the Amazon Secrets Manager documentation. After creating the secret, keep the Secret name, secretName for the next step.

   • When selecting Key/value pairs, create a pair for the key credentials with the value credentialString.

5. In the Amazon Glue Data Catalog, create a connection by following the steps in https://docs.amazonaws.cn/glue/latest/dg/console-connections.html. After creating the connection, keep the connection name, connectionName, for the next step.

   • When selecting a Connection type, select Google BigQuery.

   • When selecting an Amazon Secret, provide secretName.

6. Grant the IAM role associated with your Amazon Glue job permission to read secretName.

7. In your Amazon Glue job configuration, provide connectionName as an Additional network connection.