Creating a BigQuery source node - Amazon Glue
Prerequisites neededAdding a BigQuery data source
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Creating a BigQuery source node

Prerequisites needed

  • A BigQuery type Amazon Glue Data Catalog connection

  • An Amazon Secrets Manager secret for your Google BigQuery credentials, used by the connection.

  • Appropriate permissions on your job to read the secret used by the connection.

  • The name and dataset of the table and corresponding Google Cloud project you would like to read.

Adding a BigQuery data source

To add a Data source – BigQuery node:

  1. Choose the connection for your BigQuery data source. Since you have created it, it should be available in the dropdown. If you need to create a connection, choose Create BigQuery connection. For more information, see Overview of using connectors and connections .

    Once you have chosen a connection, you can view the connection properties by clicking View properties.

  2. Identify what BigQuery data you would like to read, then choose a BigQuery Source option

    • Choose a single table – allows you to pull all data from a table.

    • Enter a custom query – allows you to customize which data is retrieved by providing a query.

  3. Describe the data you would like to read

    (Required) set Parent Project to the project containing your table, or a billing parent project, if relevant.

    If you chose a single table, set Table to the name of a Google BigQuery table in the following format: [dataset].[table]

    If you chose a query, provide it to Query. In your query, refer to tables with their fully qualified table name, in the format: [project].[dataset].[tableName].

  4. Provide BigQuery properties

    If you chose a single table, you do not need to provide additional properties.

    If you chose a query, you must provide the following Custom Google BigQuery properties:

    • Set viewsEnabled to true.

    • Set materializationDataset to a dataset. The GCP principal authenticated by the credentials provided through the Amazon Glue connection must be able to create tables in this dataset.