Amazon Glue 用SDK于 Java 2.x 的示例 - Amazon SDK for Java 2.x
Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅 中国的 Amazon Web Services 服务入门 (PDF)

本文属于机器翻译版本。若本译文内容与英语原文存在差异,则一律以英文原文为准。

Amazon Glue 用SDK于 Java 2.x 的示例

以下代码示例向您展示了如何使用with来执行操作和实现常见场景 Amazon Glue。 Amazon SDK for Java 2.x

基础知识是向您展示如何在服务中执行基本操作的代码示例。

操作是大型程序的代码摘录,必须在上下文中运行。您可以通过操作了解如何调用单个服务函数,还可以通过函数相关场景的上下文查看操作。

每个示例都包含一个指向完整源代码的链接,您可以在其中找到有关如何在上下文中设置和运行代码的说明。

开始使用

以下代码示例展示了如何开始使用 Amazon Glue。

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

package com.example.glue; import software.amazon.awssdk.regions.Region; import software.amazon.awssdk.services.glue.GlueClient; import software.amazon.awssdk.services.glue.model.ListJobsRequest; import software.amazon.awssdk.services.glue.model.ListJobsResponse; import java.util.List; public class HelloGlue { public static void main(String[] args) { GlueClient glueClient = GlueClient.builder() .region(Region.US_EAST_1) .build(); listJobs(glueClient); } public static void listJobs(GlueClient glueClient) { ListJobsRequest request = ListJobsRequest.builder() .maxResults(10) .build(); ListJobsResponse response = glueClient.listJobs(request); List<String> jobList = response.jobNames(); jobList.forEach(job -> { System.out.println("Job Name: " + job); }); } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 ListJobs” 中的。

基础知识

以下代码示例展示了如何:

  • 创建一个爬虫来抓取公有 Amazon S3 存储桶并生成格式CSV元数据的数据库。

  • 列出您的中的数据库和表的相关信息 Amazon Glue Data Catalog。

  • 创建任务以从 S3 存储桶中提取CSV数据、转换数据,并将JSON格式化的输出加载到另一个 S3 存储桶中。

  • 列出有关作业运行的信息,查看转换后的数据,并清除资源。

有关更多信息,请参阅教程: Amazon Glue Studio 入门

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Before running this Java V2 code example, set up your development * environment, including your credentials. * <p> * For more information, see the following documentation topic: * * https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/get-started.html * * To set up the resources, see this documentation topic: * * https://docs.aws.amazon.com/glue/latest/ug/tutorial-add-crawler.html * * This example performs the following tasks: * * 1. Create a database. * 2. Create a crawler. * 3. Get a crawler. * 4. Start a crawler. * 5. Get a database. * 6. Get tables. * 7. Create a job. * 8. Start a job run. * 9. List all jobs. * 10. Get job runs. * 11. Delete a job. * 12. Delete a database. * 13. Delete a crawler. */ public class GlueScenario { public static final String DASHES = new String(new char[80]).replace("\0", "-"); public static void main(String[] args) throws InterruptedException { final String usage = """ Usage: <iam> <s3Path> <cron> <dbName> <crawlerName> <jobName> <scriptLocation> <locationUri> <bucketNameSc>\s Where: iam - The ARN of the IAM role that has AWS Glue and S3 permissions.\s s3Path - The Amazon Simple Storage Service (Amazon S3) target that contains data (for example, s3://<bucket name>/read). cron - A cron expression used to specify the schedule (i.e., cron(15 12 * * ? *). dbName - The database name.\s crawlerName - The name of the crawler.\s jobName - The name you assign to this job definition. scriptLocation - The Amazon S3 path to a script that runs a job. locationUri - The location of the database (you can find this file in resources folder). bucketNameSc - The Amazon S3 bucket name used when creating a job """; if (args.length != 9) { System.out.println(usage); return; } Scanner scanner = new Scanner(System.in); String iam = args[0]; String s3Path = args[1]; String cron = args[2]; String dbName = args[3]; String crawlerName = args[4]; String jobName = args[5]; String scriptLocation = args[6]; String locationUri = args[7]; String bucketNameSc = args[8]; Region region = Region.US_EAST_1; GlueClient glueClient = GlueClient.builder() .region(region) .build(); System.out.println(DASHES); System.out.println("Welcome to the AWS Glue scenario."); System.out.println(""" AWS Glue is a fully managed extract, transform, and load (ETL) service provided by Amazon Web Services (AWS). It is designed to simplify the process of building, running, and maintaining ETL pipelines, which are essential for data integration and data warehousing tasks. One of the key features of AWS Glue is its ability to automatically discover and catalog data stored in various sources, such as Amazon S3, Amazon RDS, Amazon Redshift, and other databases. This cataloging process creates a central metadata repository, known as the AWS Glue Data Catalog, which provides a unified view of an organization's data assets. This metadata can then be used to create ETL jobs, which can be scheduled and run on-demand or on a regular basis. Lets get started. """); waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("1. Create a database."); try { createDatabase(glueClient, dbName, locationUri); } catch (GlueException e) { if (e.awsErrorDetails().errorMessage().equals("Database already exists.")) { System.out.println("Database " + dbName + " already exists. Skipping creation."); } else { System.err.println(e.awsErrorDetails().errorMessage()); return; } } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("2. Create a crawler."); try { createGlueCrawler(glueClient, iam, s3Path, cron, dbName, crawlerName); } catch (GlueException e) { if (e.awsErrorDetails().errorMessage().contains("already exists")) { System.out.println("Crawler " + crawlerName + " already exists. Skipping creation."); } else { System.err.println(e.awsErrorDetails().errorMessage()); System.exit(1); } } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("3. Get a crawler."); try { getSpecificCrawler(glueClient, crawlerName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("4. Start a crawler."); try { startSpecificCrawler(glueClient, crawlerName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("5. Get a database."); try { getSpecificDatabase(glueClient, dbName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("*** Wait 5 min for the tables to become available"); TimeUnit.MINUTES.sleep(5); System.out.println("6. Get tables."); String myTableName; try { myTableName = getGlueTables(glueClient, dbName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("7. Create a job."); try { createJob(glueClient, jobName, iam, scriptLocation); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("8. Start a Job run."); try { startJob(glueClient, jobName, dbName, myTableName, bucketNameSc); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("9. List all jobs."); try { getAllJobs(glueClient); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("10. Get job runs."); try { getJobRuns(glueClient, jobName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("11. Delete a job."); try { deleteJob(glueClient, jobName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } System.out.println("*** Wait 5 MIN for the " + crawlerName + " to stop"); TimeUnit.MINUTES.sleep(5); waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("12. Delete a database."); try { deleteDatabase(glueClient, dbName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("Delete a crawler."); try { deleteSpecificCrawler(glueClient, crawlerName); } catch (GlueException e) { System.err.println(e.awsErrorDetails().errorMessage()); return; } waitForInputToContinue(scanner); System.out.println(DASHES); System.out.println(DASHES); System.out.println("Successfully completed the AWS Glue Scenario"); System.out.println(DASHES); } /** * Creates a Glue database with the specified name and location URI. * * @param glueClient The Glue client to use for the database creation. * @param dbName The name of the database to create. * @param locationUri The location URI for the database. */ public static void createDatabase(GlueClient glueClient, String dbName, String locationUri) { try { DatabaseInput input = DatabaseInput.builder() .description("Built with the AWS SDK for Java V2") .name(dbName) .locationUri(locationUri) .build(); CreateDatabaseRequest request = CreateDatabaseRequest.builder() .databaseInput(input) .build(); glueClient.createDatabase(request); System.out.println(dbName + " was successfully created"); } catch (GlueException e) { throw e; } } /** * Creates a new AWS Glue crawler using the AWS Glue Java API. * * @param glueClient the AWS Glue client used to interact with the AWS Glue service * @param iam the IAM role that the crawler will use to access the data source * @param s3Path the S3 path that the crawler will scan for data * @param cron the cron expression that defines the crawler's schedule * @param dbName the name of the AWS Glue database where the crawler will store the metadata * @param crawlerName the name of the crawler to be created */ public static void createGlueCrawler(GlueClient glueClient, String iam, String s3Path, String cron, String dbName, String crawlerName) { try { S3Target s3Target = S3Target.builder() .path(s3Path) .build(); List<S3Target> targetList = new ArrayList<>(); targetList.add(s3Target); CrawlerTargets targets = CrawlerTargets.builder() .s3Targets(targetList) .build(); CreateCrawlerRequest crawlerRequest = CreateCrawlerRequest.builder() .databaseName(dbName) .name(crawlerName) .description("Created by the AWS Glue Java API") .targets(targets) .role(iam) .schedule(cron) .build(); glueClient.createCrawler(crawlerRequest); System.out.println(crawlerName + " was successfully created"); } catch (GlueException e) { throw e; } } /** * Retrieves a specific crawler from the AWS Glue service and waits for it to be in the "READY" state. * * @param glueClient the AWS Glue client used to interact with the Glue service * @param crawlerName the name of the crawler to be retrieved */ public static void getSpecificCrawler(GlueClient glueClient, String crawlerName) throws InterruptedException { try { GetCrawlerRequest crawlerRequest = GetCrawlerRequest.builder() .name(crawlerName) .build(); boolean ready = false; while (!ready) { GetCrawlerResponse response = glueClient.getCrawler(crawlerRequest); String status = response.crawler().stateAsString(); if (status.compareTo("READY") == 0) { ready = true; } Thread.sleep(3000); } System.out.println("The crawler is now ready"); } catch (GlueException | InterruptedException e) { throw e; } } /** * Starts a specific AWS Glue crawler. * * @param glueClient the AWS Glue client to use for the crawler operation * @param crawlerName the name of the crawler to start * @throws GlueException if there is an error starting the crawler */ public static void startSpecificCrawler(GlueClient glueClient, String crawlerName) { try { StartCrawlerRequest crawlerRequest = StartCrawlerRequest.builder() .name(crawlerName) .build(); glueClient.startCrawler(crawlerRequest); System.out.println(crawlerName + " was successfully started!"); } catch (GlueException e) { throw e; } } /** * Retrieves the specific database from the AWS Glue service. * * @param glueClient an instance of the AWS Glue client used to interact with the service * @param databaseName the name of the database to retrieve * @throws GlueException if there is an error retrieving the database from the AWS Glue service */ public static void getSpecificDatabase(GlueClient glueClient, String databaseName) { try { GetDatabaseRequest databasesRequest = GetDatabaseRequest.builder() .name(databaseName) .build(); GetDatabaseResponse response = glueClient.getDatabase(databasesRequest); Instant createDate = response.database().createTime(); // Convert the Instant to readable date. DateTimeFormatter formatter = DateTimeFormatter.ofLocalizedDateTime(FormatStyle.SHORT) .withLocale(Locale.US) .withZone(ZoneId.systemDefault()); formatter.format(createDate); System.out.println("The create date of the database is " + createDate); } catch (GlueException e) { throw e; } } /** * Retrieves the names of the tables in the specified Glue database. * * @param glueClient the Glue client to use for the operation * @param dbName the name of the Glue database to retrieve the table names from * @return the name of the first table retrieved, or an empty string if no tables were found */ public static String getGlueTables(GlueClient glueClient, String dbName) { String myTableName = ""; try { GetTablesRequest tableRequest = GetTablesRequest.builder() .databaseName(dbName) .build(); GetTablesResponse response = glueClient.getTables(tableRequest); List<Table> tables = response.tableList(); if (tables.isEmpty()) { System.out.println("No tables were returned"); } else { for (Table table : tables) { myTableName = table.name(); System.out.println("Table name is: " + myTableName); } } } catch (GlueException e) { throw e; } return myTableName; } /** * Starts a job run in AWS Glue. * * @param glueClient the AWS Glue client to use for the job run * @param jobName the name of the Glue job to run * @param inputDatabase the name of the input database * @param inputTable the name of the input table * @param outBucket the URL of the output S3 bucket * @throws GlueException if there is an error starting the job run */ public static void startJob(GlueClient glueClient, String jobName, String inputDatabase, String inputTable, String outBucket) { try { Map<String, String> myMap = new HashMap<>(); myMap.put("--input_database", inputDatabase); myMap.put("--input_table", inputTable); myMap.put("--output_bucket_url", outBucket); StartJobRunRequest runRequest = StartJobRunRequest.builder() .workerType(WorkerType.G_1_X) .numberOfWorkers(10) .arguments(myMap) .jobName(jobName) .build(); StartJobRunResponse response = glueClient.startJobRun(runRequest); System.out.println("The request Id of the job is " + response.responseMetadata().requestId()); } catch (GlueException e) { throw e; } } /** * Creates a new AWS Glue job. * * @param glueClient the AWS Glue client to use for the operation * @param jobName the name of the job to create * @param iam the IAM role to associate with the job * @param scriptLocation the location of the script to be used by the job * @throws GlueException if there is an error creating the job */ public static void createJob(GlueClient glueClient, String jobName, String iam, String scriptLocation) { try { JobCommand command = JobCommand.builder() .pythonVersion("3") .name("glueetl") .scriptLocation(scriptLocation) .build(); CreateJobRequest jobRequest = CreateJobRequest.builder() .description("A Job created by using the AWS SDK for Java V2") .glueVersion("2.0") .workerType(WorkerType.G_1_X) .numberOfWorkers(10) .name(jobName) .role(iam) .command(command) .build(); glueClient.createJob(jobRequest); System.out.println(jobName + " was successfully created."); } catch (GlueException e) { throw e; } } /** * Retrieves and prints information about all the jobs in the Glue data catalog. * * @param glueClient the Glue client used to interact with the AWS Glue service */ public static void getAllJobs(GlueClient glueClient) { try { GetJobsRequest jobsRequest = GetJobsRequest.builder() .maxResults(10) .build(); GetJobsResponse jobsResponse = glueClient.getJobs(jobsRequest); List<Job> jobs = jobsResponse.jobs(); for (Job job : jobs) { System.out.println("Job name is : " + job.name()); System.out.println("The job worker type is : " + job.workerType().name()); } } catch (GlueException e) { throw e; } } /** * Retrieves the job runs for a given Glue job and prints the status of the job runs. * * @param glueClient the Glue client used to make API calls * @param jobName the name of the Glue job to retrieve the job runs for */ public static void getJobRuns(GlueClient glueClient, String jobName) { try { GetJobRunsRequest runsRequest = GetJobRunsRequest.builder() .jobName(jobName) .maxResults(20) .build(); boolean jobDone = false; while (!jobDone) { GetJobRunsResponse response = glueClient.getJobRuns(runsRequest); List<JobRun> jobRuns = response.jobRuns(); for (JobRun jobRun : jobRuns) { String jobState = jobRun.jobRunState().name(); if (jobState.compareTo("SUCCEEDED") == 0) { System.out.println(jobName + " has succeeded"); jobDone = true; } else if (jobState.compareTo("STOPPED") == 0) { System.out.println("Job run has stopped"); jobDone = true; } else if (jobState.compareTo("FAILED") == 0) { System.out.println("Job run has failed"); jobDone = true; } else if (jobState.compareTo("TIMEOUT") == 0) { System.out.println("Job run has timed out"); jobDone = true; } else { System.out.println("*** Job run state is " + jobRun.jobRunState().name()); System.out.println("Job run Id is " + jobRun.id()); System.out.println("The Glue version is " + jobRun.glueVersion()); } TimeUnit.SECONDS.sleep(5); } } } catch (GlueException e) { throw e; } catch (InterruptedException e) { throw new RuntimeException(e); } } /** * Deletes a Glue job. * * @param glueClient the Glue client to use for the operation * @param jobName the name of the job to be deleted * @throws GlueException if there is an error deleting the job */ public static void deleteJob(GlueClient glueClient, String jobName) { try { DeleteJobRequest jobRequest = DeleteJobRequest.builder() .jobName(jobName) .build(); glueClient.deleteJob(jobRequest); System.out.println(jobName + " was successfully deleted"); } catch (GlueException e) { throw e; } } /** * Deletes a AWS Glue Database. * * @param glueClient An instance of the AWS Glue client used to interact with the AWS Glue service. * @param databaseName The name of the database to be deleted. * @throws GlueException If an error occurs while deleting the database. */ public static void deleteDatabase(GlueClient glueClient, String databaseName) { try { DeleteDatabaseRequest request = DeleteDatabaseRequest.builder() .name(databaseName) .build(); glueClient.deleteDatabase(request); System.out.println(databaseName + " was successfully deleted"); } catch (GlueException e) { throw e; } } /** * Deletes a specific AWS Glue crawler. * * @param glueClient the AWS Glue client object * @param crawlerName the name of the crawler to be deleted * @throws GlueException if an error occurs during the deletion process */ public static void deleteSpecificCrawler(GlueClient glueClient, String crawlerName) { try { DeleteCrawlerRequest deleteCrawlerRequest = DeleteCrawlerRequest.builder() .name(crawlerName) .build(); glueClient.deleteCrawler(deleteCrawlerRequest); System.out.println(crawlerName + " was deleted"); } catch (GlueException e) { throw e; } } private static void waitForInputToContinue(Scanner scanner) { while (true) { System.out.println(""); System.out.println("Enter 'c' followed by <ENTER> to continue:"); String input = scanner.nextLine(); if (input.trim().equalsIgnoreCase("c")) { System.out.println("Continuing with the program..."); System.out.println(""); break; } else { // Handle invalid input. System.out.println("Invalid input. Please try again."); } } } }

操作

以下代码示例显示了如何使用CreateCrawler

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Creates a new AWS Glue crawler using the AWS Glue Java API. * * @param glueClient the AWS Glue client used to interact with the AWS Glue service * @param iam the IAM role that the crawler will use to access the data source * @param s3Path the S3 path that the crawler will scan for data * @param cron the cron expression that defines the crawler's schedule * @param dbName the name of the AWS Glue database where the crawler will store the metadata * @param crawlerName the name of the crawler to be created */ public static void createGlueCrawler(GlueClient glueClient, String iam, String s3Path, String cron, String dbName, String crawlerName) { try { S3Target s3Target = S3Target.builder() .path(s3Path) .build(); List<S3Target> targetList = new ArrayList<>(); targetList.add(s3Target); CrawlerTargets targets = CrawlerTargets.builder() .s3Targets(targetList) .build(); CreateCrawlerRequest crawlerRequest = CreateCrawlerRequest.builder() .databaseName(dbName) .name(crawlerName) .description("Created by the AWS Glue Java API") .targets(targets) .role(iam) .schedule(cron) .build(); glueClient.createCrawler(crawlerRequest); System.out.println(crawlerName + " was successfully created"); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 CreateCrawler” 中的。

以下代码示例显示了如何使用CreateJob

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Creates a new AWS Glue job. * * @param glueClient the AWS Glue client to use for the operation * @param jobName the name of the job to create * @param iam the IAM role to associate with the job * @param scriptLocation the location of the script to be used by the job * @throws GlueException if there is an error creating the job */ public static void createJob(GlueClient glueClient, String jobName, String iam, String scriptLocation) { try { JobCommand command = JobCommand.builder() .pythonVersion("3") .name("glueetl") .scriptLocation(scriptLocation) .build(); CreateJobRequest jobRequest = CreateJobRequest.builder() .description("A Job created by using the AWS SDK for Java V2") .glueVersion("2.0") .workerType(WorkerType.G_1_X) .numberOfWorkers(10) .name(jobName) .role(iam) .command(command) .build(); glueClient.createJob(jobRequest); System.out.println(jobName + " was successfully created."); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 CreateJob” 中的。

以下代码示例显示了如何使用DeleteCrawler

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Deletes a specific AWS Glue crawler. * * @param glueClient the AWS Glue client object * @param crawlerName the name of the crawler to be deleted * @throws GlueException if an error occurs during the deletion process */ public static void deleteSpecificCrawler(GlueClient glueClient, String crawlerName) { try { DeleteCrawlerRequest deleteCrawlerRequest = DeleteCrawlerRequest.builder() .name(crawlerName) .build(); glueClient.deleteCrawler(deleteCrawlerRequest); System.out.println(crawlerName + " was deleted"); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 DeleteCrawler” 中的。

以下代码示例显示了如何使用DeleteDatabase

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Deletes a AWS Glue Database. * * @param glueClient An instance of the AWS Glue client used to interact with the AWS Glue service. * @param databaseName The name of the database to be deleted. * @throws GlueException If an error occurs while deleting the database. */ public static void deleteDatabase(GlueClient glueClient, String databaseName) { try { DeleteDatabaseRequest request = DeleteDatabaseRequest.builder() .name(databaseName) .build(); glueClient.deleteDatabase(request); System.out.println(databaseName + " was successfully deleted"); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 DeleteDatabase” 中的。

以下代码示例显示了如何使用DeleteJob

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Deletes a Glue job. * * @param glueClient the Glue client to use for the operation * @param jobName the name of the job to be deleted * @throws GlueException if there is an error deleting the job */ public static void deleteJob(GlueClient glueClient, String jobName) { try { DeleteJobRequest jobRequest = DeleteJobRequest.builder() .jobName(jobName) .build(); glueClient.deleteJob(jobRequest); System.out.println(jobName + " was successfully deleted"); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 DeleteJob” 中的。

以下代码示例显示了如何使用GetCrawler

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Retrieves a specific crawler from the AWS Glue service and waits for it to be in the "READY" state. * * @param glueClient the AWS Glue client used to interact with the Glue service * @param crawlerName the name of the crawler to be retrieved */ public static void getSpecificCrawler(GlueClient glueClient, String crawlerName) throws InterruptedException { try { GetCrawlerRequest crawlerRequest = GetCrawlerRequest.builder() .name(crawlerName) .build(); boolean ready = false; while (!ready) { GetCrawlerResponse response = glueClient.getCrawler(crawlerRequest); String status = response.crawler().stateAsString(); if (status.compareTo("READY") == 0) { ready = true; } Thread.sleep(3000); } System.out.println("The crawler is now ready"); } catch (GlueException | InterruptedException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 GetCrawler” 中的。

以下代码示例显示了如何使用GetDatabase

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Retrieves the specific database from the AWS Glue service. * * @param glueClient an instance of the AWS Glue client used to interact with the service * @param databaseName the name of the database to retrieve * @throws GlueException if there is an error retrieving the database from the AWS Glue service */ public static void getSpecificDatabase(GlueClient glueClient, String databaseName) { try { GetDatabaseRequest databasesRequest = GetDatabaseRequest.builder() .name(databaseName) .build(); GetDatabaseResponse response = glueClient.getDatabase(databasesRequest); Instant createDate = response.database().createTime(); // Convert the Instant to readable date. DateTimeFormatter formatter = DateTimeFormatter.ofLocalizedDateTime(FormatStyle.SHORT) .withLocale(Locale.US) .withZone(ZoneId.systemDefault()); formatter.format(createDate); System.out.println("The create date of the database is " + createDate); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 GetDatabase” 中的。

以下代码示例显示了如何使用GetJobRuns

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Retrieves the job runs for a given Glue job and prints the status of the job runs. * * @param glueClient the Glue client used to make API calls * @param jobName the name of the Glue job to retrieve the job runs for */ public static void getJobRuns(GlueClient glueClient, String jobName) { try { GetJobRunsRequest runsRequest = GetJobRunsRequest.builder() .jobName(jobName) .maxResults(20) .build(); boolean jobDone = false; while (!jobDone) { GetJobRunsResponse response = glueClient.getJobRuns(runsRequest); List<JobRun> jobRuns = response.jobRuns(); for (JobRun jobRun : jobRuns) { String jobState = jobRun.jobRunState().name(); if (jobState.compareTo("SUCCEEDED") == 0) { System.out.println(jobName + " has succeeded"); jobDone = true; } else if (jobState.compareTo("STOPPED") == 0) { System.out.println("Job run has stopped"); jobDone = true; } else if (jobState.compareTo("FAILED") == 0) { System.out.println("Job run has failed"); jobDone = true; } else if (jobState.compareTo("TIMEOUT") == 0) { System.out.println("Job run has timed out"); jobDone = true; } else { System.out.println("*** Job run state is " + jobRun.jobRunState().name()); System.out.println("Job run Id is " + jobRun.id()); System.out.println("The Glue version is " + jobRun.glueVersion()); } TimeUnit.SECONDS.sleep(5); } } } catch (GlueException e) { throw e; } catch (InterruptedException e) { throw new RuntimeException(e); } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 GetJobRuns” 中的。

以下代码示例显示了如何使用GetTables

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Retrieves the names of the tables in the specified Glue database. * * @param glueClient the Glue client to use for the operation * @param dbName the name of the Glue database to retrieve the table names from * @return the name of the first table retrieved, or an empty string if no tables were found */ public static String getGlueTables(GlueClient glueClient, String dbName) { String myTableName = ""; try { GetTablesRequest tableRequest = GetTablesRequest.builder() .databaseName(dbName) .build(); GetTablesResponse response = glueClient.getTables(tableRequest); List<Table> tables = response.tableList(); if (tables.isEmpty()) { System.out.println("No tables were returned"); } else { for (Table table : tables) { myTableName = table.name(); System.out.println("Table name is: " + myTableName); } } } catch (GlueException e) { throw e; } return myTableName; }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 GetTables” 中的。

以下代码示例显示了如何使用StartCrawler

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Starts a specific AWS Glue crawler. * * @param glueClient the AWS Glue client to use for the crawler operation * @param crawlerName the name of the crawler to start * @throws GlueException if there is an error starting the crawler */ public static void startSpecificCrawler(GlueClient glueClient, String crawlerName) { try { StartCrawlerRequest crawlerRequest = StartCrawlerRequest.builder() .name(crawlerName) .build(); glueClient.startCrawler(crawlerRequest); System.out.println(crawlerName + " was successfully started!"); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 StartCrawler” 中的。

以下代码示例显示了如何使用StartJobRun

SDK适用于 Java 2.x
注意

还有更多相关信息 GitHub。查找完整示例,学习如何在 Amazon 代码示例存储库中进行设置和运行。

/** * Starts a job run in AWS Glue. * * @param glueClient the AWS Glue client to use for the job run * @param jobName the name of the Glue job to run * @param inputDatabase the name of the input database * @param inputTable the name of the input table * @param outBucket the URL of the output S3 bucket * @throws GlueException if there is an error starting the job run */ public static void startJob(GlueClient glueClient, String jobName, String inputDatabase, String inputTable, String outBucket) { try { Map<String, String> myMap = new HashMap<>(); myMap.put("--input_database", inputDatabase); myMap.put("--input_table", inputTable); myMap.put("--output_bucket_url", outBucket); StartJobRunRequest runRequest = StartJobRunRequest.builder() .workerType(WorkerType.G_1_X) .numberOfWorkers(10) .arguments(myMap) .jobName(jobName) .build(); StartJobRunResponse response = glueClient.startJobRun(runRequest); System.out.println("The request Id of the job is " + response.responseMetadata().requestId()); } catch (GlueException e) { throw e; } }
  • 有关API详细信息,请参阅 “Amazon SDK for Java 2.x API参考 StartJobRun” 中的。