AWS Elastic Beanstalk
Developer Guide (API Version 2010-12-01)
AWS services or capabilities described in AWS documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon AWS.

Using Elastic Beanstalk with Amazon CloudWatch Logs

With CloudWatch Logs, you can monitor and archive your Elastic Beanstalk application, system, and custom log files. Furthermore, you can configure alarms that make it easier for you to take actions in response to specific log stream events that your metric filters extract. The CloudWatch Logs agent installed on each Amazon EC2 in your environment publishes metric data points to the CloudWatch service for each log group you configure. Each log group applies its own filter patterns to determine what log stream events to send to CloudWatch as data points. Log streams that belong to the same log group share the same retention, monitoring, and access control settings. You can configure Elastic Beanstalk to automatically stream logs to the CloudWatch service, as described in Streaming CloudWatch Logs. For more information about CloudWatch Logs, including terminology and concepts, go to Monitoring System, Application, and Custom Log Files.

The following figure displays graphs on the Monitoring page for an environment that is configured with CloudWatch Logs integration. The example metrics in this environment are named CWLHttp4xx and CWLHttp5xx. In the image, the CWLHttp4xx metric has triggered an alarm according to conditions specified in the configuration files.

The following figure displays graphs on the Alarms page for the example alarms named AWSEBCWLHttp4xxPercentAlarm and AWSEBCWLHttp5xxCountAlarm that correspond to the CWLHttp4xx and CWLHttp5xx metrics, respectively.

Streaming CloudWatch Logs

Since Elastic Beanstalk can spin up Amazon EC2 instances on demand (provided you have enabled that feature), Elastic Beanstalk provides another option so that you can stream log entries from those Amazon EC2 instances to CloudWatch. To enable this feature, select Enabled for Log streaming, set Retention to the number of days to save the logs, and select the Lifecycle setting for whether the logs are saved after the instance is terminated, as shown in the following figure, which saves the logs for 7 days and keeps the logs after terminating the instance. You can also enable these settings using the eb logs command. Note that this feature is only available in containers since this release.

Note also that once you enable CloudWatch logs, you'll see the View in CloudWatch Console link. Click that link to see your CloudWatch logs in the CloudWatch console.

Note

If you do not have the AWSElasticBeanstalkWebTier or AWSElasticBeanstalkWorkerTier Elastic Beanstalk managed policy in your Elastic Beanstalk instance profile, you must add the following to your profile to enable this feature.

{
  "Version": "2012-10-17",
  "Statement": [
  {
    "Effect": "Allow",
    "Action": [
      "logs:PutLogEvents",
      "logs:CreateLogStream"
    ],
    "Resource": [
    "arn:aws:logs:*:*:log-group:/aws/elasticbeanstalk*"
    ]
  }
  ]
}

Elastic Beanstalk installs a CloudWatch log agent with the default configuration settings on each instance it creates. Learn more at CloudWatch Logs Agent Reference.

Different containers stream different logs. The following table lists the logs, by container.

Container

Logs

Java

  • /var/log/eb-activity.log

  • /var/log/nginx/access.log

  • /var/log/nginx/error.log

  • /var/log/web-1.error.log

  • /var/log/web-1.log

Node.js

  • /var/log/eb-activity.log

  • /var/log/nodejs/nodejs.log

  • /var/log/nginx/error.log

  • /var/log/nginx/access.log

  • /var/log/httpd/error.log

  • /var/log/httpd/access.log

Php

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

Python

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

  • /opt/python/log/supervisord.log

Ruby (Puma)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/puma/puma.log

  • /var/log/nginx/access.log

Ruby (passenger)

  • /var/log/eb-activity.log

  • /var/app/support/logs/passenger.log

  • /var/app/support/logs/access.log

  • /var/app/support/logs/error.log

Tomcat

  • /var/log/eb-activity.log

  • /var/log/httpd/error_log

  • /var/log/httpd/access_log

  • /var/log/nginx/error_log

  • /var/log/nginx/access_log

Go

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/nginx/access.log

Docker

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

Multi-Docker(generic)

  • /var/log/eb-activity.log

  • /var/log/ecs/ecs-init.log

  • /var/log/eb-ecs-mgr.log

  • /var/log/ecs/ecs-agent.log

  • /var/log/docker-events.log

Glass fish (Preconfigured docker)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

Go (Preconfigured docker)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

Python (Preconfigured docker)

  • /var/log/eb-activity.log

  • /var/log/nginx/error.log

  • /var/log/docker-events.log

  • /var/log/docker

  • /var/log/nginx/access.log

You can also enable CloudWatch logs using the eb logs --cloudwatch enable command.

Setting Up CloudWatch Logs Integration with Configuration Files

When you create or update an environment, you can use the sample configuration files in the following list to set up and configure integration with CloudWatch Logs. You can include the .zip file that contains following configuration files or the extracted configuration files in the .ebextensions directory at the top level of your application source bundle. Use the appropriate files for the web server for your container type. For more information about the web server used by each container type, see Elastic Beanstalk Supported Platforms.

Before you can configure integration with CloudWatch Logs using configuration files, you must set up IAM permissions to use with the CloudWatch Logs agent. You can attach the following custom policy to the instance profile that you assign to your environment:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:GetLogEvents",
        "logs:PutLogEvents",
        "logs:DescribeLogGroups",
        "logs:DescribeLogStreams",
        "logs:PutRetentionPolicy"
      ],
      "Resource": [
        "arn:aws:logs:us-west-2:*:*"
      ]
    }
  ]
}

Replace the region in the above policy with the region in which you launch your environment.

Note

You cannot configure CloudWatch Logs integration with Elastic Beanstalk applications created in .NET containers.

You can download the configuration files at the following locations:

Each .zip file contains the following configuration files:

  • cwl-setup.config – This file installs the CloudWatch Logs agent on each Amazon EC2 instance in your environment and then configures the agent. This file also creates the general.conf file when Elastic Beanstalk launches the instance. You can use the cwl-setup.config file without any modifications.

    If you prefer, you can manually set up the CloudWatch Logs agent on a new instance as explained in either Quick Start: Install and Configure the CloudWatch Logs Agent on a New EC2 Instance (for new instances) or Quick Start: Install and Configure the CloudWatch Logs Agent on an Existing EC2 Instance (for existing instances) in the Amazon CloudWatch Developer Guide.

  • cwl-webrequest-metrics.config – This file specifies which logs the CloudWatch Logs agent monitors. The file also specifies the metric filters the CloudWatch Logs agent applies to each log that it monitors. Metric filters include filter patterns that map to the space-delimited entries in your log files. (If you have custom logs, update or replace the example filter patterns in this example configuration file as needed.)

    Metric filters also include metric transformations that specify what metric name and value to use when the CloudWatch Logs agent sends metric data points to the CloudWatch service. The CloudWatch Logs agent sends metric data points based on whether any entries in the web server access log file match the filter patterns.

    Finally, the configuration file also includes an alarm action to send a message to an Amazon Simple Notification Service topic, if you created one for your environment, when the alarm conditions specified in the cwl-setup.config file are met. For more information about filter patterns, see Filter and Pattern Syntax in the Amazon CloudWatch Developer Guide. For more information about Amazon SNS, go to the Amazon Simple Notification Service Developer Guide. For more information about managing alarms from the Elastic Beanstalk management console, see Manage Alarms.

    Note

    CloudWatch costs are applied to your AWS account for any alarms that you use.

  • eb-logs.config – This file sets up the CloudWatch Logs log files for the CloudWatch Logs agent. This configuration file also ensures that log files are copied to Amazon S3 as part of log rotation. You can use this file without any modifications.

Troubleshooting CloudWatch Logs Integration

If Elastic Beanstalk cannot launch your environment when you try to integrate Elastic Beanstalk with CloudWatch Logs, you can investigate the following common issues:

  • Your IAM role lacks the required IAM permissions.

  • You attempted to launch an environment in a region where CloudWatch Logs is not supported.

  • Access logs do not exist at the path specified in the cwl-webrequest-metrics.config file (/var/log/httpd/elasticbeanstalk-access_log).