Getting started with the Amazon Glue Data Catalog
The Amazon Glue Data Catalog is your persistent technical metadata store. It is a managed service that you can use to store, annotate, and share metadata in the Amazon Cloud. For more information, see Amazon Glue Data Catalog.
The Amazon Glue console and some user interfaces were recently updated. |
Overview
You can use this tutorial to create your first Amazon Glue Data Catalog, which uses an Amazon S3 bucket as your data source.
In this tutorial, you'll do the following using the Amazon Glue console:
-
Create a database
-
Create a table
-
Use an Amazon S3 bucket as a data source
After completing these steps, you will have successfully used an Amazon S3 bucket as the data source to populate the Amazon Glue Data Catalog.
Step 1: Create a database
To get started, sign in to the Amazon Web Services Management Console and open the
Amazon Glue console
To create a database using the Amazon Glue console:
-
In the Amazon Glue console, choose Databases under Data catalog from the left-hand menu.
-
Choose Add database.
-
In the Create a database page, enter a name for the database. In the Location - optional section, set the URI location for use by clients of the Data Catalog. If you don't know this, you can continue with creating the database.
-
(Optional). Enter a description for the database.
-
Choose Create database.
Congratulations, you've just set up your first database using the Amazon Glue console. Your new database will appear in the list of available databases. You can edit the database by choosing the database's name from the Databases dashboard.
Next steps
Other ways to create a database:
You just created a database using the Amazon Glue console, but there are other ways to create a database:
-
You can use crawlers to create a database and tables for you automatically. To set up a database using crawlers, see Working with Crawlers in the Amazon Glue Console.
-
You can use Amazon CloudFormation templates. See Creating Amazon Glue Resources Using Amazon Glue Data Catalog Templates.
-
You can also create a database using the Amazon Glue Database API operations.
To create a database using the
create
operation, structure the request by including theDatabaseInput
(required) parameters.For example:
-
The following are examples of how you can use the CLI, Boto3, or DDL to define a table based on the same flights_data.csv file from the S3 bucket that you used in the tutorial.
For more information about the Database API data types, structure, and operations, see Database API.
Next Steps
In the next section, you'll create a table and add that table to your database.
You can also explore the settings and permissions for your Data Catalog. See Working with Data Catalog Settings in the Amazon Glue Console.
Step 2. Create a table
In this step, you create a table using the Amazon Glue console.
-
In the Amazon Glue console, choose Tables in the left-hand menu.
-
Choose Add table.
-
Set your table's properties by entering a name for your table in Table details.
-
In the Databases section, choose the database that you created in Step 1 from the drop-down menu.
-
In Add a data store section, S3 will be selected by default as the type of source.
-
For Data is located in , choose Specified path in another account.
-
Copy and paste the path for the Include path input field:
s3://crawler-public-us-west-2/flight/2016/csv/
-
In the section Data format, for Classification, choose CSV. and for Delimiter, choose comma (,). Choose Next.
-
You are asked to define a schema. A schema defines the structure and format of a data record. Choose Add column. (For more information, see See Schema registries).
-
Specify the column properties:
-
Enter a column name.
-
For Column type, 'string' is already selected by default.
-
For Column number, '1' is already selected by default.
-
Choose Add.
-
-
You are asked to add partition indexes. This is optional. To skip this step, choose Next.
-
A summary of the table properties is displayed. If everything looks as expected, choose Create. Otherwise, choose Back and make edits as needed.
Congratulations, you've successfully created a table manually and associated it to a database. Your newly created table will appear in the Tables dashboard. From the dashboard, you can modify and manage all your tables.
For more information, see Working with Tables in the Amazon Glue Console.
Next steps
Next steps
Now that the Data Catalog is populated, you can begin authoring jobs in Amazon Glue. See Building visual ETL jobs with Amazon Glue Studio.
In addition to using the console, there are other ways to define tables in the Data Catalog including:
-
Using the Amazon CLI
, Boto3, or data definition language (DDL) -
The following are examples of how you can use the CLI, Boto3, or DDL to define a table based on the same flights_data.csv file from the S3 bucket that you used in the tutorial.
See the documentation on how to structure an Amazon CLI command. The CLI example contains the JSON syntax for the 'aws glue create-table --table-input' value.