监控数据加载 - Amazon Aurora
Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异,请参阅 中国的 Amazon Web Services 服务入门 (PDF)

监控数据加载

Aurora PostgreSQL Limitless Database 提供了多种监控数据加载任务的方式:

列出数据加载任务

您可以连接到集群端点,然后使用 rds_aurora.limitless_data_load_jobs 视图列出数据加载任务。

postgres_limitless=> SELECT * FROM rds_aurora.limitless_data_load_jobs LIMIT 6; job_id | status | message | source_db_identifier | source_db_name | full_load_complete_time | progress_details | start_time | last_updated_time | streaming_mode | source_engine_type | ignore_primary_key_conflict | is_dryrun ---------------+-----------+---------+-------------------------------+----------------+-------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------+------------------------+------------------------+-------------------+--------------------+-----------------------------+----------- 1725697520693 | COMPLETED | | persistent-kdm-auto-source-01 | postgres | 2024-09-07 08:48:15+00 | {"FULL_LOAD": {"STATUS": "COMPLETED", "DETAILS": "9 of 9 tables loaded", "COMPLETED_AT": "2024/09/07 08:48:15+00", "RECORDS_MIGRATED": 600003}} | 2024-09-07 08:47:13+00 | 2024-09-07 08:48:15+00 | full_load | aurora_postgresql | t | f 1725696114225 | COMPLETED | | persistent-kdm-auto-source-01 | postgres | 2024-09-07 08:24:20+00 | {"FULL_LOAD": {"STATUS": "COMPLETED", "DETAILS": "3 of 3 tables loaded", "COMPLETED_AT": "2024/09/07 08:24:20+00", "RECORDS_MIGRATED": 200001}} | 2024-09-07 08:23:56+00 | 2024-09-07 08:24:20+00 | full_load | aurora_postgresql | t | f 1725696067630 | COMPLETED | | persistent-kdm-auto-source-01 | postgres | 2024-09-07 08:23:45+00 | {"FULL_LOAD": {"STATUS": "COMPLETED", "DETAILS": "6 of 6 tables loaded", "COMPLETED_AT": "2024/09/07 08:23:45+00", "RECORDS_MIGRATED": 400002}} | 2024-09-07 08:23:10+00 | 2024-09-07 08:23:45+00 | full_load | aurora_postgresql | t | f 1725694221753 | CANCELED | | persistent-kdm-auto-source-01 | postgres | | {} | 2024-09-07 07:31:18+00 | 2024-09-07 07:51:49+00 | full_load_and_cdc | aurora_postgresql | t | f 1725691698210 | COMPLETED | | persistent-kdm-auto-source-01 | postgres | 2024-09-07 07:10:51+00 | {"FULL_LOAD": {"STATUS": "COMPLETED", "DETAILS": "1 of 1 tables loaded", "COMPLETED_AT": "2024/09/07 07:10:51+00", "RECORDS_MIGRATED": 100000}} | 2024-09-07 07:10:42+00 | 2024-09-07 07:10:52+00 | full_load | aurora_postgresql | t | f 1725691695049 | COMPLETED | | persistent-kdm-auto-source-01 | postgres | 2024-09-07 07:10:48+00 | {"FULL_LOAD": {"STATUS": "COMPLETED", "DETAILS": "1 of 1 tables loaded", "COMPLETED_AT": "2024/09/07 07:10:48+00", "RECORDS_MIGRATED": 100000}} | 2024-09-07 07:10:41+00 | 2024-09-07 07:10:48+00 | full_load | aurora_postgresql | t | f (6 rows)

任务记录将在 90 天后删除。

使用任务 ID 查看数据加载任务的详细信息

如果您知道任务 ID,则可以连接到集群端点,然后使用 rds_aurora.limitless_data_load_job_details 视图查看该数据加载任务的详细信息,包括表名称、任务状态和已加载的行数。您可以在对数据加载启动函数的响应中获取任务 ID,也可以从 rds_aurora.limitless_data_load_jobs 视图中获取。

postgres_limitless=> SELECT * FROM rds_aurora.limitless_data_load_job_details WHERE job_id='1725696114225'; job_id | destination_table_name | destination_schema_name | start_time | status | full_load_rows | full_load_total_rows | full_load_complete_time | cdc_insert | cdc_update | cdc_delete --------------+------------------------+-------------------------+------------------------+-----------+----------------+----------------------+-------------------------+------------+------------+------------ 1725696114225 | standard_1 | public | 2024-09-07 08:23:57+00 | COMPLETED | 100000 | 100000 | 2024-09-07 08:24:08+00 | 0 | 0 | 0 1725696114225 | standard_2 | public | 2024-09-07 08:24:08+00 | COMPLETED | 100000 | 100000 | 2024-09-07 08:24:17+00 | 0 | 0 | 0 1725696114225 | standard_3 | public | 2024-09-07 08:24:18+00 | COMPLETED | 1 | 1 | 2024-09-07 08:24:20+00 | 0 | 0 | 0 1725696114225 | standard_4 | public | 2024-09-07 08:23:58+00 | PENDING | 0 | 0 | | 0 | 0 | 0 (4 rows)

任务记录将在 90 天后删除。

监控 Amazon CloudWatch 日志组

数据加载任务状态更改为 RUNNING 后,您可以使用 Amazon CloudWatch Logs 查看运行进度。

监控 CloudWatch 日志流

登录 Amazon Web Services Management Console并打开 CloudWatch 控制台(https://console.aws.amazon.com/cloudwatch/)。

  1. 导航到日志,然后导航到日志组

  2. 选择 /aws/rds/aurora-limitless-database 日志组。

  3. 通过 job_id 搜索您的数据加载任务的日志流。

    日志流的模式为 Data-Load-Job-job_id

  4. 选择日志流以查看日志事件。

每个日志流都显示包含任务状态和已加载到 Aurora PostgreSQL Limitless Database 目标表中的行数的事件。如果数据加载任务失败,还会创建错误日志,显示失败状态和原因。

任务记录将在 90 天后删除。

监控 RDS 事件

数据加载任务还会发布 RDS 事件,例如当任务成功、失败或取消时。您可以查看目标数据库中的事件。

有关更多信息,请参阅 数据库分片组事件