Tutorial for scraping Redis OSS Prometheus metrics on Amazon ECS Fargate
This tutorial provides a hands-on introduction to scrape the Prometheus metrics of a sample Redis OSS application in an Amazon ECS Fargate cluster. The Redis OSS Prometheus exporter target will be auto-discovered by the CloudWatch agent with Prometheus metric support based on the container’s docker labels.
Redis OSS (https://redis.io/) is an open source (BSD licensed), in-memory data structure
store, used as a database, cache and message broker. For more information, see redis
redis_exporter (MIT License licensed) is used to expose the Redis OSS prometheus metrics
on the specified port (default: 0.0.0.0:9121). For more information, see redis_exporter
The Docker images in the following two Docker Hub repositories are used in this tutorial:
Prerequisite
To collect metrics from a sample Prometheus workload for Amazon ECS, you must be running Container Insights in the cluster. For information about installing Container Insights, see Setting up Container Insights on Amazon ECS.
Topics
Set the Amazon ECS Fargate cluster environment variable
To set the Amazon ECS Fargate cluster environment variable
-
Install the Amazon ECS CLI if you haven't already done so. For more information, see Installing the Amazon ECS CLI.
-
Set the new Amazon ECS cluster name and Region. For example:
ECS_CLUSTER_NAME=ecs-fargate-redis-tutorial AWS_DEFAULT_REGION=ca-central-1
-
(Optional) If you don't already have an Amazon ECS Fargate cluster where you want to install the sample Redis OSS workload and CloudWatch agent, you can create one by entering the following command.
ecs-cli up --capability-iam \ --cluster $ECS_CLUSTER_NAME \ --launch-type FARGATE \ --region $AWS_DEFAULT_REGION
The expected result of this command is as follows:
INFO[0000] Created cluster cluster=ecs-fargate-redis-tutorial region=ca-central-1 INFO[0001] Waiting for your cluster resources to be created... INFO[0001] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS VPC created: vpc-xxxxxxxxxxxxxxxxx Subnet created: subnet-xxxxxxxxxxxxxxxxx Subnet created: subnet-xxxxxxxxxxxxxxxxx Cluster creation succeeded.
Set the network environment variables for the Amazon ECS Fargate cluster
To set the network environment variables for the Amazon ECS Fargate cluster
-
Set your VPC and subnet ID of the Amazon ECS cluster. If you created a new cluster in the previous procedure, you'll see these values in the result of the final command. Otherwise, use the IDs of the existing cluster that you are going to use with Redis.
ECS_CLUSTER_VPC=
vpc-xxxxxxxxxxxxxxxxx
ECS_CLUSTER_SUBNET_1=subnet-xxxxxxxxxxxxxxxxx
ECS_CLUSTER_SUBNET_2=subnet-xxxxxxxxxxxxxxxxx
-
In this tutorial, we are going to install the Redis OSS application and the CloudWatch agent in the default security group of the Amazon ECS cluster’s VPC. The default security group allows all network connection within the same security group so the CloudWatch agent can scrape the Prometheus metrics exposed on the Redis OSS containers. In a real production environment, you might want to create dedicated security groups for the Redis OSS application and CloudWatch agent and set customized permissions for them.
Enter the following command to get the default security group ID.
aws ec2 describe-security-groups \ --filters Name=vpc-id,Values=$ECS_CLUSTER_VPC \ --region $AWS_DEFAULT_REGION
Then set the Fargate cluster deafult security group variable by entering the following command, replacing
my-default-security-group
with the value you found from the previous command.ECS_CLUSTER_SECURITY_GROUP=
my-default-security-group
Install the sample Redis OSS workload
To install the sample Redis OSS workload which exposes the Prometheus metrics
-
Download the Redis OSS Amazon CloudFormation template by entering the following command.
curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/sample_traffic/redis/redis-traffic-sample.yaml
-
Set the IAM role names to be created for Redis OSS by entering the following commands.
REDIS_ECS_TASK_ROLE_NAME=redis-prometheus-demo-ecs-task-role-name REDIS_ECS_EXECUTION_ROLE_NAME=redis-prometheus-demo-ecs-execution-role-name
-
Install the sample Redis OSS workload by entering the following command.
aws cloudformation create-stack --stack-name Redis-Prometheus-Demo-ECS-$ECS_CLUSTER_NAME-fargate-awsvpc \ --template-body file://redis-traffic-sample.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \ ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET_1 \ ParameterKey=TaskRoleName,ParameterValue=$REDIS_ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$REDIS_ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region $AWS_DEFAULT_REGION
The Amazon CloudFormation stack creates four resources:
-
One ECS task role
-
One ECS task execution role
-
One Redis OSS task definition
-
One Redis OSS service
In the Redis OSS task definition, two containers are defined:
-
The primary container runs a simple Redis OSS application and opens port 6379 for access.
-
The other container runs the Redis OSS exporter process to expose the Prometheus metrics on port 9121. This is the container to be discovered and scraped by the CloudWatch agent. The following docker label is defined so that the CloudWatch agent can discover this container based on it.
ECS_PROMETHEUS_EXPORTER_PORT: 9121
Configure the CloudWatch agent to scrape Redis OSS Prometheus metrics
To configure the CloudWatch agent to scrape Redis OSS Prometheus metrics
-
Download the latest version of
cwagent-ecs-prometheus-metric-for-awsvpc.yaml
by entering the following command.curl -O https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/ecs-task-definition-templates/deployment-mode/replica-service/cwagent-prometheus/cloudformation-quickstart/cwagent-ecs-prometheus-metric-for-awsvpc.yaml
-
Open the file with a text editor, and find the full CloudWatch agent configuration behind the
value
key in theresource:CWAgentConfigSSMParameter
section.Then, in the
ecs_service_discovery
section shown here, thedocker_label
-based service discovery is enabled with the default settings which are based onECS_PROMETHEUS_EXPORTER_PORT
, which matches the docker label we defined in the Redis OSS ECS task definition. So we do not need to make any changes in this section:ecs_service_discovery": { "sd_frequency": "1m", "sd_result_file": "/tmp/cwagent_ecs_auto_sd.yaml", * "docker_label": { },* ...
For the
metric_declaration
section, the default setting does not allow any Redis OSS metrics. Add the following section to allow Redis OSS metrics. Be sure to follow the existing indentation pattern.{ "source_labels": ["container_name"], "label_matcher": "^redis-exporter-.*$", "dimensions": [["ClusterName","TaskDefinitionFamily"]], "metric_selectors": [ "^redis_net_(in|out)put_bytes_total$", "^redis_(expired|evicted)_keys_total$", "^redis_keyspace_(hits|misses)_total$", "^redis_memory_used_bytes$", "^redis_connected_clients$" ] }, { "source_labels": ["container_name"], "label_matcher": "^redis-exporter-.*$", "dimensions": [["ClusterName","TaskDefinitionFamily","cmd"]], "metric_selectors": [ "^redis_commands_total$" ] }, { "source_labels": ["container_name"], "label_matcher": "^redis-exporter-.*$", "dimensions": [["ClusterName","TaskDefinitionFamily","db"]], "metric_selectors": [ "^redis_db_keys$" ] },
-
If you already have the CloudWatch agent deployed in the Amazon ECS cluster by Amazon CloudFormation, you can create a change set by entering the following commands.
ECS_LAUNCH_TYPE=FARGATE CREATE_IAM_ROLES=True ECS_CLUSTER_SUBNET=$ECS_CLUSTER_SUBNET_1 ECS_TASK_ROLE_NAME=
your_selected_ecs_task_role_name
ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
aws cloudformation create-change-set --stack-name CWAgent-Prometheus-ECS-$ECS_CLUSTER_NAME-$ECS_LAUNCH_TYPE-awsvpc \ --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSLaunchType,ParameterValue=$ECS_LAUNCH_TYPE \ ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \ ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION} \ --change-set-name redis-scraping-support Open the Amazon CloudFormation console at https://console.amazonaws.cn/cloudformation
. -
Review the newly created changeset
redis-scraping-support
. You should see one change applied to theCWAgentConfigSSMParameter
resource. Execute the changeset and restart the CloudWatch agent task by entering the following commands.aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 0 \ --service cwagent-prometheus-replica-service-$ECS_LAUNCH_TYPE-awsvpc \ --region ${AWS_DEFAULT_REGION}
-
Wait about 10 seconds, and then enter the following command.
aws ecs update-service --cluster $ECS_CLUSTER_NAME \ --desired-count 1 \ --service cwagent-prometheus-replica-service-$ECS_LAUNCH_TYPE-awsvpc \ --region ${AWS_DEFAULT_REGION}
-
If you are installing the CloudWatch agent with Prometheus metric collecting for the cluster for the first time, please enter the following commands:
ECS_LAUNCH_TYPE=FARGATE CREATE_IAM_ROLES=True ECS_CLUSTER_SUBNET=$ECS_CLUSTER_SUBNET_1 ECS_TASK_ROLE_NAME=
your_selected_ecs_task_role_name
ECS_EXECUTION_ROLE_NAME=your_selected_ecs_execution_role_name
aws cloudformation create-stack --stack-name CWAgent-Prometheus-ECS-$ECS_CLUSTER_NAME-$ECS_LAUNCH_TYPE-awsvpc \ --template-body file://cwagent-ecs-prometheus-metric-for-awsvpc.yaml \ --parameters ParameterKey=ECSClusterName,ParameterValue=$ECS_CLUSTER_NAME \ ParameterKey=CreateIAMRoles,ParameterValue=$CREATE_IAM_ROLES \ ParameterKey=ECSLaunchType,ParameterValue=$ECS_LAUNCH_TYPE \ ParameterKey=SecurityGroupID,ParameterValue=$ECS_CLUSTER_SECURITY_GROUP \ ParameterKey=SubnetID,ParameterValue=$ECS_CLUSTER_SUBNET \ ParameterKey=TaskRoleName,ParameterValue=$ECS_TASK_ROLE_NAME \ ParameterKey=ExecutionRoleName,ParameterValue=$ECS_EXECUTION_ROLE_NAME \ --capabilities CAPABILITY_NAMED_IAM \ --region ${AWS_DEFAULT_REGION}
Viewing your Redis OSS metrics
This tutorial sends the following metrics to the ECS/ContainerInsights/Prometheus namespace in CloudWatch. You can use the CloudWatch console to see the metrics in that namespace.
Metric Name | Dimensions |
---|---|
|
ClusterName, |
|
ClusterName, |
|
ClusterName, |
|
ClusterName, |
|
ClusterName, |
|
ClusterName, |
|
ClusterName, |
|
ClusterName, |
|
|
|
|
Note
The value of the cmd dimension can be: append
,
client
, command
, config
,
dbsize
, flushall
, get
, incr
,
info
, latency
, or slowlog
.
The value of the db dimension can be db0
to
db15
.
You can also create a CloudWatch dashboard for your Redis OSS Prometheus metrics.
To create a dashboard for Redis OSS Prometheus metrics
-
Create environment variables, replacing the values below to match your deployment.
DASHBOARD_NAME=
your_cw_dashboard_name
ECS_TASK_DEF_FAMILY=redis-prometheus-demo-$ECS_CLUSTER_NAME-fargate-awsvpc -
Enter the following command to create the dashboard.
curl https://raw.githubusercontent.com/aws-samples/amazon-cloudwatch-container-insights/latest/k8s-deployment-manifest-templates/deployment-mode/service/cwagent-prometheus/sample_cloudwatch_dashboards/redis/cw_dashboard_redis.json \ | sed "s/{{YOUR_AWS_REGION}}/${REGION_NAME}/g" \ | sed "s/{{YOUR_CLUSTER_NAME}}/${CLUSTER_NAME}/g" \ | sed "s/{{YOUR_NAMESPACE}}/${NAMESPACE}/g" \