Configuring a crawler

Name may contain letters (A-Z), numbers (0-9), hyphens (-), or underscores (_), and can be up to 255 characters long.

Descriptions can be up to 2048 characters long.

Use tags to organize and identify your resources. For more information, see the following:

Select the appropriate option for Is your data already mapped to Amazon Glue tables? choose 'Not yet' or 'Yes'. By default, 'Not yet' is selected.

The crawler can access data stores directly as the source of the crawl, or it can use existing tables in the Data Catalog as the source. If the crawler uses existing catalog tables, it crawls the data stores that are specified by those catalog tables.

Select or add the list of data sources to be scanned by the crawler.

(Optional) If you choose JDBC as the data source, you can use your own JDBC drivers when specifying the Connection access where the driver info is stored.

When evaluating what to include or exclude in a crawl, a crawler starts by evaluating the required include path. For Amazon S3, MongoDB, MongoDB Atlas, Amazon DocumentDB (with MongoDB compatibility), and relational data stores, you must specify an include path.

For MongoDB, MongoDB Atlas, and Amazon DocumentDB (with MongoDB compatibility), the syntax is database/collection.

For JDBC data stores, the syntax is either database-name/schema-name/table-name or database-name/table-name. The syntax depends on whether the database engine supports schemas within a database. For example, for database engines such as MySQL or Oracle, don't specify a schema-name in your include path. You can substitute the percent sign (%) for a schema or table in the include path to represent all schemas or all tables in a database. You cannot substitute the percent sign (%) for database in the include path.

Defines the maximum depth of the Amazon S3 path that the crawler can traverse to discover the Iceberg or Hudi metadata folder in your Amazon S3 path. The purpose of this parameter is to limit the crawler run time. The default value is 10 and the maximum is 20.

These enable you to exclude certain files or tables from the crawl. The exclude path is relative to the include path. For example, to exclude a table in your JDBC data store, type the table name in the exclude path.

A crawler connects to a JDBC data store using an Amazon Glue connection that contains a JDBC URI connection string. The crawler only has access to objects in the database engine using the JDBC user name and password in the Amazon Glue connection. The crawler can only create tables that it can access through the JDBC connection. After the crawler accesses the database engine with the JDBC URI, the include path is used to determine which tables in the database engine are created in the Data Catalog. For example, with MySQL, if you specify an include path of MyDatabase/%, then all tables within MyDatabase are created in the Data Catalog. When accessing Amazon Redshift, if you specify an include path of MyDatabase/%, then all tables within all schemas for database MyDatabase are created in the Data Catalog. If you specify an include path of MyDatabase/MySchema/%, then all tables in database MyDatabase and schema MySchema are created.

After you specify an include path, you can then exclude objects from the crawl that your include path would otherwise include by specifying one or more Unix-style glob exclude patterns. These patterns are applied to your include path to determine which objects are excluded. These patterns are also stored as a property of tables created by the crawler. Amazon Glue PySpark extensions, such as create_dynamic_frame.from_catalog, read the table properties and exclude objects defined by the exclude pattern.

Amazon Glue supports the following kinds of glob patterns in the exclude pattern.

Amazon Glue interprets glob exclude patterns as follows:

/mybucket/myfolder/
   departments/
      finance.json
      market-us.json
      market-emea.json
      market-ap.json
   employees/
      hr.json
      john.csv
      jane.csv
      juan.txt

 2015/01/{[!0],0[8-9]}**, 2015/0[2-9]/**, 2015/1[0-2]/**    
 

MyDatabase/MySchema/
   HR_us
   HR_fr
   Employees_Table
   Finance
   Market_US_Table
   Market_EMEA_Table
   Market_AP_Table

Each source type requires a different set of additional parameters. The following is an incomplete list:

Select or add an Amazon Glue connection. For information about connections, see Connecting to data.

Select additional metadata properties for the crawler to crawl.

Type a custom JDBC driver class name for the crawler to connect to the data source:

Choose an existing Amazon S3 path to a .jar file. This is where the .jar file will be stored when using a custom JDBC driver for the crawler to connect to the data source.

Select whether to crawl a data sample only. If not selected the entire table is crawled. Scanning all the records can take a long time when the table is not a high throughput table.

Select how you want to create the Delta Lake tables:

Specify the percentage of the DynamoDB table Read Capacity Units to use by the crawler. Read capacity units is a term defined by DynamoDB, and is a numeric value that acts as rate limiter for the number of reads that can be performed on that table per second. Enter a value between 0.1 and 1.5. If not specified, defaults to 0.5% for provisioned tables and 1/4 of maximum configured capacity for on-demand tables. Note that only provisioned capacity mode should be used with Amazon Glue crawlers.

Optionally include a Network connection to use with this Amazon S3 target. Note that each crawler is limited to one Network connection so any other Amazon S3 targets will also use the same connection (or none, if left blank).

For information about connections, see Connecting to data.

Specify the number of files in each leaf folder to be crawled when crawling sample files in a dataset. When this feature is turned on, instead of crawling all the files in this dataset, the crawler randomly selects some files in each leaf folder to crawl.

The sampling crawler is best suited for customers who have previous knowledge about their data formats and know that schemas in their folders do not change. Turning on this feature will significantly reduce crawler runtime.

A valid value is an integer between 1 and 249. If not specified, all the files are crawled.

This field is a global field that affects all Amazon S3 data sources.

Define custom classifiers before defining crawlers. A classifier checks whether a given file is in a format the crawler can handle. If it is, the classifier creates a schema in the form of a StructType object that matches that data format.

For more information, see Adding classifiers to a crawler in Amazon Glue.

The crawler assumes this role. It must have permissions similar to the Amazon managed policy AWSGlueServiceRole. For Amazon S3 and DynamoDB sources, it must also have permissions to access the data store. If the crawler reads Amazon S3 data encrypted with Amazon Key Management Service (Amazon KMS), then the role must have decrypt permissions on the Amazon KMS key.

For an Amazon S3 data store, additional permissions attached to the role would be similar to the following:

{
   "Version": "2012-10-17",
    "Statement": [
        {
          "Effect": "Allow",
          "Action": [
              "s3:GetObject",
              "s3:PutObject"
          ],
          "Resource": [
              "arn:aws-cn:s3:::bucket/object*"
          ]
        }
    ]
}

For an Amazon DynamoDB data store, additional permissions attached to the role would be similar to the following:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "dynamodb:DescribeTable",
        "dynamodb:Scan"
      ],
      "Resource": [
        "arn:aws-cn:dynamodb:region:account-id:table/table-name*"
      ]
    }
  ]
}

In order to add your own JDBC driver, additional permissions need to be added.

Example Amazon S3 policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-name/driver-parent-folder/driver.jar",
                "arn:aws:s3:::bucket-name"
            ]
        }
    ]
}
                            
                        

Amazon Glue creates the following folders (_crawler and _glue_job_crawler at the same level as the JDBC driver in your Amazon S3 bucket. For example, if the driver path is <s3-path/driver_folder/driver.jar>, then the following folders will be created if they do not already exist:

Optionally, you can add a security configuration to a crawler to specify at-rest encryption options.

For more information, see Step 2: Create an IAM role for Amazon Glue and Identity and access management for Amazon Glue.

Allow the crawler to use Lake Formation credentials for crawling the data source.

Checking Use Lake Formation credentials for crawling S3 data source will allow the crawler to use Lake Formation credentials for crawling the data source. If the data source belongs to another account, you must provide the registered account ID. Otherwise, the crawler will crawl only those data sources associated to the account. Only applicable to Amazon S3 and Data Catalog data sources.

Settings include security configurations. For more information, see the following:

Options include how the crawler should handle detected schema changes, deleted objects in the data store, and more. For more information, see Customizing crawler behavior

You can run a crawler on demand or define a time-based schedule for your crawlers and jobs in Amazon Glue. The definition of these schedules uses the Unix-like cron syntax. For more information, see Scheduling an Amazon Glue crawler.