创建 Amazon OpenSearch Ingestion 管道

OpenSearch Ingestion 运行异步进程来构建管道。当管道状态为 Active 时，您可以开始将数据提取到管道。

create-pipeline 命令接受以字符串形式或在 .yaml 文件中的管道配置。如果您以字符串形式提供配置，则必须使用 \n 转义每一个新行。例如，"log-pipeline:\n source:\n http:\n processor:\n - grok:\n ...

以下示例命令采用以下配置创建管道：

aws osis create-pipeline \
  --pipeline-name my-pipeline \
  --min-units 4 \
  --max-units 10 \
  --log-publishing-options  IsLoggingEnabled=true,CloudWatchLogDestination={LogGroup="MyLogGroup"} \
  --vpc-options SecurityGroupIds={sg-12345678,sg-9012345},SubnetIds=subnet-1212234567834asdf \
  --pipeline-configuration-body "file://pipeline-config.yaml"

OpenSearch Ingestion 运行异步进程来构建管道。当管道状态为 Active 时，您可以开始将数据提取到管道。要检查管道的状态，请使用GetPipeline命令。

要使用 OpenSearch Ingestion API 创建 OpenSearch 摄取管道，请调用该操作。CreatePipeline

最初创建管道后，当 OpenSearch Ingestion 为采集数据做准备时，它会经历多个阶段。要查看管道创建的各个阶段，请选择管道名称以查看其管道设置页面。在状态下，选择查看详细信息。

管道要经过以下几个阶段才可以摄取数据：

使用get-pipeline-change-progress命令检查管道的状态。以下 Amazon CLI 请求检查名为的管道的状态my-pipeline：

aws osis get-pipeline-change-progress \
    --pipeline-name my-pipeline

响应：

{
   "ChangeProgressStatuses": {
      "ChangeProgressStages": [ 
         { 
            "Description": "Validating pipeline configuration",
            "LastUpdated": 1.671055851E9,
            "Name": "VALIDATION",
            "Status": "PENDING"
         }
      ],
      "StartTime": 1.671055851E9,
      "Status": "PROCESSING",
      "TotalNumberOfStages": 5
   }
}

要使用 OpenSearch Ingestion API 跟踪管道创建的状态，请调用该操作。GetPipelineChangeProgress

要使用获取所有可用蓝图的列表 Amazon CLI，请发送list-pipeline-blueprints请求。

aws osis list-pipeline-blueprints 

该请求返回所有可用蓝图的列表。

要获取有关特定蓝图的更多详细信息，请使用以下get-pipeline-blueprint命令：

aws osis get-pipeline-blueprint --blueprint-name AWS-ApacheLogPipeline

该请求返回 Apache 日志管道蓝图的内容：

{
   "Blueprint":{
      "PipelineConfigurationBody":"###\n  # Limitations: https://docs.aws.amazon.com/opensearch-service/latest/ingestion/ingestion.html#ingestion-limitations\n###\n###\n  # apache-log-pipeline:\n    # This pipeline receives logs via http (e.g. FluentBit), extracts important values from the logs by matching\n    # the value in the 'log' key against the grok common Apache log pattern. The grokked logs are then sent\n    # to OpenSearch to an index named 'logs'\n###\n\nversion: \"2\"\napache-log-pipeline:\n  source:\n    http:\n      # Provide the path for ingestion. ${pipelineName} will be replaced with pipeline name configured for this pipeline.\n      # In this case it would be \"/apache-log-pipeline/logs\". This will be the FluentBit output URI value.\n      path: \"/${pipelineName}/logs\"\n  processor:\n    - grok:\n        match:\n          log: [ \"%{COMMONAPACHELOG_DATATYPED}\" ]\n  sink:\n    - opensearch:\n        # Provide an AWS OpenSearch Service domain endpoint\n        # hosts: [ \"https://search-mydomain-1a2a3a4a5a6a7a8a9a0a9a8a7a.us-east-1.es.amazonaws.com\" ]\n        aws:\n          # Provide a Role ARN with access to the domain. This role should have a trust relationship with osis-pipelines.amazonaws.com\n          # sts_role_arn: \"arn:aws:iam::123456789012:role/Example-Role\"\n          # Provide the region of the domain.\n          # region: \"us-east-1\"\n          # Enable the 'serverless' flag if the sink is an Amazon OpenSearch Serverless collection\n          # serverless: true\n        index: \"logs\"\n        # Enable the S3 DLQ to capture any failed requests in an S3 bucket\n        # dlq:\n          # s3:\n            # Provide an S3 bucket\n            # bucket: \"your-dlq-bucket-name\"\n            # Provide a key path prefix for the failed requests\n            # key_path_prefix: \"${pipelineName}/logs/dlq\"\n            # Provide the region of the bucket.\n            # region: \"us-east-1\"\n            # Provide a Role ARN with access to the bucket. This role should have a trust relationship with osis-pipelines.amazonaws.com\n            # sts_role_arn: \"arn:aws:iam::123456789012:role/Example-Role\"\n",
      "BlueprintName":"AWS-ApacheLogPipeline"
   }
}

要使用 OpenSearch Ingestion API 获取有关管道蓝图的信息，请使用和操作。ListPipelineBlueprintsGetPipelineBlueprint