Creating a BigQuery connection - Amazon Glue
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Creating a BigQuery connection

To connect to Google BigQuery from Amazon Glue, you will need to create and store your Google Cloud Platform credentials in a Amazon Secrets Manager secret, then associate that secret with a Google BigQuery Amazon Glue connection.

To configure a connection to BigQuery:
  1. In Google Cloud Platform, create and identify relevant resources:

  2. In Google Cloud Platform, create and export service account credentials:

    You can use the BigQuery credentials wizard to expedite this step: Create credentials.

    To create a service account in GCP, follow the tutorial available in Create service accounts.

    • When selecting project, select the project containing your BigQuery table.

    • When selecting GCP IAM roles for your service account, add or create a role that would grant appropriate permissions to run BigQuery jobs to read, write or create BigQuery tables.

    To create credentials for your service account, follow the tutorial available in Create a service account key.

    • When selecting key type, select JSON.

    You should now have downloaded a JSON file with credentials for your service account. It should look similar to the following:

    { "type": "service_account", "project_id": "*****", "private_key_id": "*****", "private_key": "*****", "client_email": "*****", "client_id": "*****", "auth_uri": "", "token_uri": "", "auth_provider_x509_cert_url": "", "client_x509_cert_url": "*****", "universe_domain": "" }
  3. base64 encode your downloaded credentials file. On an Amazon CloudShell session or similar, you can do this from the command line by running cat credentialsFile.json | base64 -w 0. Retain the output of this command, credentialString.

  4. In Amazon Secrets Manager, create a secret using your Google Cloud Platform credentials. To create a secret in Secrets Manager, follow the tutorial available in Create an Amazon Secrets Manager secret in the Amazon Secrets Manager documentation. After creating the secret, keep the Secret name, secretName for the next step.

    • When selecting Key/value pairs, create a pair for the key credentials with the value credentialString.

  5. In the Amazon Glue Data Catalog, create a connection by following the steps in After creating the connection, keep the connection name, connectionName, for the next step.

    • When selecting a Connection type, select Google BigQuery.

    • When selecting an Amazon Secret, provide secretName.

  6. Grant the IAM role associated with your Amazon Glue job permission to read secretName.

  7. In your Amazon Glue job configuration, provide connectionName as an Additional network connection.