Fargate logging

To configure the log router

In the following steps, replace every example value with your own values.

1. Create a dedicated Kubernetes namespace named aws-observability.

   1. Save the following contents to a file named aws-observability-namespace.yaml on your computer. The value for name must be aws-observability and the aws-observability: enabled label is required.

      kind: Namespace
      apiVersion: v1
      metadata:
        name: aws-observability
        labels:
          aws-observability: enabled

   2. Create the namespace.

      kubectl apply -f aws-observability-namespace.yaml
      

2. Create a ConfigMap with a Fluent Conf data value to ship container logs to a destination. Fluent Conf is Fluent Bit, which is a fast and lightweight log processor configuration language that's used to route container logs to a log destination of your choice. For more information, see Configuration File in the Fluent Bit documentation.

   Important

   The main sections included in a typical Fluent Conf are Service, Input, Filter, and Output. The Fargate log router however, only accepts:

   • The Filter and Output sections.

   • A Parser section.

   If you provide any other sections, they will be rejected.

   The Fargate log router manages the Service and Input sections. It has the following Input section, which can't be modified and isn't needed in your ConfigMap. However, you can get insights from it, such as the memory buffer limit and the tag applied for logs.

   [INPUT]
       Name tail
       Buffer_Max_Size 66KB
       DB /var/log/flb_kube.db
       Mem_Buf_Limit 45MB
       Path /var/log/containers/*.log
       Read_From_Head On
       Refresh_Interval 10
       Rotate_Wait 30
       Skip_Long_Lines On
       Tag kube.*

   When creating the ConfigMap, take into account the following rules that Fargate uses to validate fields:

   • [FILTER], [OUTPUT], and [PARSER] are supposed to be specified under each corresponding key. For example, [FILTER] must be under filters.conf. You can have one or more [FILTER]s under filters.conf. The [OUTPUT] and [PARSER] sections should also be under their corresponding keys. By specifying multiple [OUTPUT] sections, you can route your logs to different destinations at the same time.

   • Fargate validates the required keys for each section. Name and match are required for each [FILTER] and [OUTPUT]. Name and format are required for each [PARSER]. The keys are case-insensitive.

   • Environment variables such as ${ENV_VAR} aren't allowed in the ConfigMap.

   • The indentation has to be the same for either directive or key-value pair within each filters.conf, output.conf, and parsers.conf. Key-value pairs have to be indented more than directives.

   • Fargate validates against the following supported filters: grep, parser, record_modifier, rewrite_tag, throttle, nest, modify, and kubernetes.

   • Fargate validates against the following supported output: es, firehose, kinesis_firehose, cloudwatch, cloudwatch_logs, and kinesis.

   • At least one supported Output plugin has to be provided in the ConfigMap to enable logging. Filter and Parser aren't required to enable logging.

   You can also run Fluent Bit on Amazon EC2 using the desired configuration to troubleshoot any issues that arise from validation. Create your ConfigMap using one of the following examples.

   Important

   Amazon EKS Fargate logging doesn't support dynamic configuration of ConfigMaps. Any changes to ConfigMaps are applied to new Pods only. Changes aren't applied to existing Pods.

   Create a ConfigMap using the example for your desired log destination.

   Note

   You can also use Amazon Kinesis Data Streams for your log destination. If you use Kinesis Data Streams, make sure that the pod execution role has been granted the kinesis:PutRecords permission. For more information, see Amazon Kinesis Data Streams Permissions in the Fluent Bit: Official Manual.

   CloudWatch

   To create a ConfigMap for CloudWatch

   You have two output options when using CloudWatch:

   • An output plugin written in C

   • An output plugin written in Golang

   The following example shows you how to use the cloudwatch_logs plugin to send logs to CloudWatch.

   1. Save the following contents to a file named aws-logging-cloudwatch-configmap.yaml. Replace region-code with the Amazon Web Services Region that your cluster is in. The parameters under [OUTPUT] are required.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: aws-logging
        namespace: aws-observability
      data:
        flb_log_cw: "false"  # Set to true to ship Fluent Bit process logs to CloudWatch.
        filters.conf: |
          [FILTER]
              Name parser
              Match *
              Key_name log
              Parser crio
          [FILTER]
              Name kubernetes
              Match kube.*
              Merge_Log On
              Keep_Log Off
              Buffer_Size 0
              Kube_Meta_Cache_TTL 300s
        output.conf: |
          [OUTPUT]
              Name cloudwatch_logs
              Match   kube.*
              region region-code
              log_group_name my-logs
              log_stream_prefix from-fluent-bit-
              log_retention_days 60
              auto_create_group true
        parsers.conf: |
          [PARSER]
              Name crio
              Format Regex
              Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>P|F) (?<log>.*)$
              Time_Key    time
              Time_Format %Y-%m-%dT%H:%M:%S.%L%z

   2. Apply the manifest to your cluster.

      kubectl apply -f aws-logging-cloudwatch-configmap.yaml
      

   3. Download the CloudWatch IAM policy to your computer. You can also view the policy on GitHub.

      curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/cloudwatchlogs/permissions.json
      

   Amazon OpenSearch Service

   To create a ConfigMap for Amazon OpenSearch Service

   If you want to send logs to Amazon OpenSearch Service, you can use es output, which is a plugin written in C. The following example shows you how to use the plugin to send logs to OpenSearch.

   1. Save the following contents to a file named aws-logging-opensearch-configmap.yaml. Replace every example value with your own values.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: aws-logging
        namespace: aws-observability
      data:
        output.conf: |
          [OUTPUT]
            Name  es
            Match *
            Host  search-example-gjxdcilagiprbglqn42jsty66y.region-code.es.amazonaws.com
            Port  443
            Index example
            Type  example_type
            AWS_Auth On
            AWS_Region region-code
            tls   On

   2. Apply the manifest to your cluster.

      kubectl apply -f aws-logging-opensearch-configmap.yaml
      

   3. Download the OpenSearch IAM policy to your computer. You can also view the policy on GitHub.

      curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/amazon-elasticsearch/permissions.json
      

      Make sure that OpenSearch Dashboards' access control is configured properly. The all_access role in OpenSearch Dashboards needs to have the Fargate Pod execution role and the IAM role mapped. The same mapping must be done for the security_manager role. You can add the previous mappings by selecting Menu, then Security, then Roles, and then select the respective roles. For more information, see How do I troubleshoot CloudWatch Logs so that it streams to my Amazon ES domain?.

   Firehose

   To create a ConfigMap for Firehose

   You have two output options when sending logs to Firehose:

   • kinesis_firehose – An output plugin written in C.

   • firehose – An output plugin written in Golang.

   The following example shows you how to use the kinesis_firehose plugin to send logs to Firehose.

   1. Save the following contents to a file named aws-logging-firehose-configmap.yaml. Replace region-code with the Amazon Web Services Region that your cluster is in.

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: aws-logging
        namespace: aws-observability
      data:
        output.conf: |
          [OUTPUT]
           Name  kinesis_firehose
           Match *
           region region-code
           delivery_stream my-stream-firehose

   2. Apply the manifest to your cluster.

      kubectl apply -f aws-logging-firehose-configmap.yaml
      

   3. Download the Firehose IAM policy to your computer. You can also view the policy on GitHub.

      curl -O https://raw.githubusercontent.com/aws-samples/amazon-eks-fluent-logging-examples/mainline/examples/fargate/kinesis-firehose/permissions.json
      

3. Create an IAM policy from the policy file you downloaded in a previous step.

   aws iam create-policy --policy-name eks-fargate-logging-policy --policy-document file://permissions.json
   

4. Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace 111122223333 with your account ID. Replace AmazonEKSFargatePodExecutionRole with your Pod execution role (for more information, see Create a Fargate Pod execution role).

   aws iam attach-role-policy \
     --policy-arn arn:aws-cn:iam::111122223333:policy/eks-fargate-logging-policy \
     --role-name AmazonEKSFargatePodExecutionRole
   

1. Locate the CloudWatch log group automatically created for your Amazon EKS cluster's Fluent Bit process logs after enabling Fargate logging. It follows the format {cluster_name}-fluent-bit-logs.

2. Delete the existing CloudWatch log streams created for each Pod's process logs in the CloudWatch log group.

3. Edit the ConfigMap and set flb_log_cw: "false".

4. Restart any existing Pods in the cluster.

1. Deploy a sample Pod.

   1. Save the following contents to a file named sample-app.yaml on your computer.

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: sample-app
        namespace: same-namespace-as-your-fargate-profile
      spec:
        replicas: 3
        selector:
          matchLabels:
            app: nginx
        template:
          metadata:
            labels:
              app: nginx
          spec:
            containers:
              - name: nginx
                image: nginx:latest
                ports:
                  - name: http
                    containerPort: 80

   2. Apply the manifest to the cluster.

      kubectl apply -f sample-app.yaml
      

2. View the NGINX logs using the destination(s) that you configured in the ConfigMap.