Fargate logging
Amazon EKS on Fargate offers a built-in log router based on Fluent Bit. This means that
you don't explicitly run a Fluent Bit container as a sidecar, but Amazon runs it for
you. All that you have to do is configure the log router. The configuration happens
through a dedicated ConfigMap
that must meet the following criteria:
-
Named
aws-logging
-
Created in a dedicated namespace called
aws-observability
-
Can't exceed 5300 characters.
Once you've created the ConfigMap
, Amazon EKS on Fargate automatically
detects it and configures the log router with it. Fargate uses a version of Amazon for Fluent Bit, an upstream compliant distribution of Fluent Bit managed by Amazon. For more
information, see Amazon for Fluent Bit
The log router allows you to use the breadth of services at Amazon for log analytics and
storage. You can stream logs from Fargate directly to Amazon CloudWatch, Amazon OpenSearch Service. You can also
stream logs to destinations such as Amazon S3
Prerequisites
-
An existing Fargate profile that specifies an existing Kubernetes namespace that you deploy Fargate Pods to. For more information, see Create a Fargate profile for your cluster.
-
An existing Fargate Pod execution role. For more information, see Create a Fargate Pod execution role.
Log router configuration
To configure the log router
In the following steps, replace every
with your own
values.example value
-
Create a dedicated Kubernetes namespace named
aws-observability
.-
Save the following contents to a file named
on your computer. The value foraws-observability-namespace
.yamlname
must beaws-observability
and theaws-observability: enabled
label is required.kind: Namespace apiVersion: v1 metadata: name: aws-observability labels: aws-observability: enabled
-
Create the namespace.
kubectl apply -f
aws-observability-namespace
.yaml
-
-
Create a
ConfigMap
with aFluent Conf
data value to ship container logs to a destination. Fluent Conf is Fluent Bit, which is a fast and lightweight log processor configuration language that's used to route container logs to a log destination of your choice. For more information, see Configuration Filein the Fluent Bit documentation. Important
The main sections included in a typical
Fluent Conf
areService
,Input
,Filter
, andOutput
. The Fargate log router however, only accepts:-
The
Filter
andOutput
sections. -
A
Parser
section.
If you provide any other sections, they will be rejected.
The Fargate log router manages the
Service
andInput
sections. It has the followingInput
section, which can't be modified and isn't needed in yourConfigMap
. However, you can get insights from it, such as the memory buffer limit and the tag applied for logs.[INPUT] Name tail DB /var/log/flb_kube.db Mem_Buf_Limit 10MB Path /var/log/containers/*.log Read_From_Head On Refresh_Interval 10 Rotate_Wait 30 Skip_Long_Lines On Tag kube.*
When creating the
ConfigMap
, take into account the following rules that Fargate uses to validate fields:-
[FILTER]
,[OUTPUT]
, and[PARSER]
are supposed to be specified under each corresponding key. For example,[FILTER]
must be underfilters.conf
. You can have one or more[FILTER]
s underfilters.conf
. The[OUTPUT]
and[PARSER]
sections should also be under their corresponding keys. By specifying multiple[OUTPUT]
sections, you can route your logs to different destinations at the same time. -
Fargate validates the required keys for each section.
Name
andmatch
are required for each[FILTER]
and[OUTPUT]
.Name
andformat
are required for each[PARSER]
. The keys are case-insensitive. -
Environment variables such as
${ENV_VAR}
aren't allowed in theConfigMap
. -
The indentation has to be the same for either directive or key-value pair within each
filters.conf
,output.conf
, andparsers.conf
. Key-value pairs have to be indented more than directives. -
Fargate validates against the following supported filters:
grep
,parser
,record_modifier
,rewrite_tag
,throttle
,nest
,modify
, andkubernetes
. -
Fargate validates against the following supported output:
es
,firehose
,kinesis_firehose
,cloudwatch
,cloudwatch_logs
, andkinesis
. -
At least one supported
Output
plugin has to be provided in theConfigMap
to enable logging.Filter
andParser
aren't required to enable logging.
You can also run Fluent Bit on Amazon EC2 using the desired configuration to troubleshoot any issues that arise from validation. Create your
ConfigMap
using one of the following examples.Important
Amazon EKS Fargate logging doesn't support dynamic configuration of
ConfigMaps
. Any changes toConfigMaps
are applied to new Pods only. Changes aren't applied to existing Pods.Create a
ConfigMap
using the example for your desired log destination.Note
You can also use Amazon Kinesis Data Streams for your log destination. If you use Kinesis Data Streams, make sure that the pod execution role has been granted the
kinesis:PutRecords
permission. For more information, see Amazon Kinesis Data Streams Permissions in the Fluent Bit: Official Manual. -
-
Create an IAM policy from the policy file you downloaded in a previous step.
aws iam create-policy --policy-name
eks-fargate-logging-policy
--policy-document file://permissions.json -
Attach the IAM policy to the pod execution role specified for your Fargate profile with the following command. Replace
with your account ID. Replace111122223333
with your Pod execution role (for more information, see Create a Fargate Pod execution role).AmazonEKSFargatePodExecutionRole
aws iam attach-role-policy \ --policy-arn arn:aws-cn:iam::
111122223333
:policy/eks-fargate-logging-policy
\ --role-nameAmazonEKSFargatePodExecutionRole
Kubernetes filter support
This feature requires the following minimum Kubernetes version and platform level, or later.
Kubernetes version | Platform level |
---|---|
1.23 and later | eks.1 |
The Fluent Bit
Kubernetes filter allows you to add Kubernetes metadata to
your log files. For more information about the filter, see Kubernetes
filters.conf: | [FILTER] Name kubernetes Match kube.* Merge_Log On Buffer_Size 0 Kube_Meta_Cache_TTL
300
s
Important
-
Kube_URL
,Kube_CA_File
,Kube_Token_Command
, andKube_Token_File
are service owned configuration parameters and must not be specified. Amazon EKS Fargate populates these values. -
Kube_Meta_Cache_TTL
is the time Fluent Bit waits until it communicates with the API server for the latest metadata. IfKube_Meta_Cache_TTL
isn't specified, Amazon EKS Fargate appends a default value of 30 minutes to lessen the load on the API server.
To ship Fluent Bit process logs to your account
You can optionally ship Fluent Bit process logs to Amazon CloudWatch using
the following ConfigMap
. Shipping Fluent Bit process logs to CloudWatch
requires additional log ingestion and storage costs.
Replace
with the Amazon Web Services Region that your cluster is in.region-code
kind: ConfigMap apiVersion: v1 metadata: name: aws-logging namespace: aws-observability labels: data: # Configuration files: server, input, filters and output # ====================================================== flb_log_cw: "true" # Ships Fluent Bit process logs to CloudWatch. output.conf: | [OUTPUT] Name cloudwatch Match kube.* region
region-code
log_group_name fluent-bit-cloudwatch log_stream_prefix from-fluent-bit- auto_create_group true
The logs are in the Amazon Web Services Region that the cluster resides in under CloudWatch.
The
log group name is
and the Fluent Bit logstream name is
my-cluster
-fluent-bit-logsfluent-bit-
.podname
-pod-namespace
Note
-
The process logs are shipped only when the Fluent Bit process successfully starts. If there is a failure while starting Fluent Bit, the process logs are missed. You can only ship process logs to CloudWatch.
-
To debug shipping process logs to your account, you can apply the previous
ConfigMap
to get the process logs. Fluent Bit failing to start is usually due to yourConfigMap
not being parsed or accepted by Fluent Bit while starting.
To stop shipping Fluent Bit process logs
Shipping Fluent Bit process logs to CloudWatch requires additional log
ingestion and storage costs. To exclude process logs in an existing
ConfigMap
setup, do the following steps.
-
Locate the CloudWatch log group automatically created for your Amazon EKS cluster's Fluent Bit process logs after enabling Fargate logging. It follows the format
{cluster_name}-fluent-bit-logs
. -
Delete the existing CloudWatch log streams created for each Pod's process logs in the CloudWatch log group.
-
Edit the
ConfigMap
and setflb_log_cw: "false"
. -
Restart any existing Pods in the cluster.
Test application
-
Deploy a sample Pod.
-
Save the following contents to a file named
on your computer.sample-app
.yamlapiVersion: apps/v1 kind: Deployment metadata: name: sample-app namespace:
same-namespace-as-your-fargate-profile
spec: replicas: 3 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:latest ports: - name: http containerPort: 80 -
Apply the manifest to the cluster.
kubectl apply -f
sample-app
.yaml
-
-
View the NGINX logs using the destination(s) that you configured in the
ConfigMap
.
Size considerations
We suggest that you plan for up to 50 MB of memory for the log router. If you expect your application to generate logs at very high throughput then you should plan for up to 100 MB.
Troubleshooting
To confirm whether the logging feature is enabled or disabled for some reason, such as
an invalid ConfigMap
, and why it's invalid, check your Pod
events with
. The output might include
Pod events that clarify whether logging is enabled or not, such as
the following example output.kubectl describe pod
pod_name
[...] Annotations: CapacityProvisioned: 0.25vCPU 0.5GB Logging: LoggingDisabled: LOGGING_CONFIGMAP_NOT_FOUND kubernetes.io/psp: eks.privileged [...] Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning LoggingDisabled <unknown> fargate-scheduler Disabled logging because aws-logging configmap was not found. configmap "aws-logging" not found
The Pod events are ephemeral with a time period depending on the
settings. You can also view a Pod's annotations using
. In the
Pod annotation, there is information about whether the logging
feature is enabled or disabled and the reason.kubectl describe pod
pod-name