CloudWatch Logs Insights query syntax - Amazon CloudWatch Logs
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

CloudWatch Logs Insights query syntax

With CloudWatch Logs Insights, you use a query language to query your log groups. The query syntax supports different functions and operations that include but aren't limited to general functions, arithmetic and comparison operations, and regular expressions. Create queries that contain multiple commands. Separate commands with the pipe character (|). Create queries that contain comments. Set off comments with the hash character (#).

Note

CloudWatch Logs Insights automatically discovers fields for different log types and generates fields that start with the @ character. For more information about these fields, see Supported logs and discovered fields in the Amazon CloudWatch User Guide.

CloudWatch Logs Insights query commands

This section includes a list of the supported CloudWatch Logs Insights query commands.

Note

Not all of the example queries in this section will work on your log events. The example queries in this section are meant to show how you can format your queries using the supported CloudWatch Logs Insights query commands. You may need to format the queries to match the format of your log events. For more query examples, see Sample queries.

  • display

    Use display to show a specific field or fields in query results.

    Example: Display one field

    The code snippet shows an example of a query that uses the parse command to extract data from @message to create the ephemeral fields loggingType and loggingMessage. The query returns all log events where the values for loggingType are ERROR. display shows only the values for loggingMessage in the query results.

    fields @message | parse @message "[*] *" as loggingType, loggingMessage | filter loggingType = "ERROR" | display loggingMessage
    Tip

    Use display only once in a query. If you use display more than once in a query, the query results show the field specified in the last occurrence of display command being used.

  • fields

    Use fields to show specific fields in query results.

    Example: Display specific fields

    The code snippet shows an example of a query that returns 20 log events and displays them in descending order. The values for @timestamp and @message are shown in the query results.

    fields @timestamp, @message | sort @timestamp desc | limit 20
    Tip

    Use this command when you don't want to use display. fields supports different functions and operations for modifying field values and creating new fields that can be used in queries.

  • filter

    Use filter to get log events that match one or more conditions.

    Example: Filter log events using one condition

    The code snippet shows an example of a query that returns all log events where the value for range is greater than 3000. The query limits the results to 20 log events and sorts the logs events by @timestamp and in descending order.

    fields @timestamp, @message | filter (range>3000) | sort @timestamp desc | limit 20

    Example: Filter log events using more than one condition

    You can use the keywords and and or to combine more than one condition.

    The code snippet shows an example of a query that returns log events where the value for range is greater than 3000 and value for accountId is equal to 123456789012. The query limits the results to 20 log events and sorts the logs events by @timestamp and in descending order.

    fields @timestamp, @message | filter (range>3000 and accountId=123456789012) | sort @timestamp desc | limit 20
  • stats

    Use stats to calculate aggregate statistics with log field values.

  • sort

    Use sort to display log events in ascending (asc) or descending (desc) order.

  • limit

    Use limit to specify the number of log events that you want your query to return.

  • parse

    Use parse to extract data from a log field and create an ephemeral field that you can process in your query. parse supports both glob mode using wildcards, and regular expressions.

    You can parse nested JSON fields with a regular expression.

    Example: Parsing a nested JSON field

    The code snippet shows how to parse a JSON log event that's been flattened during ingestion.

    {'fieldsA': 'logs', 'fieldsB': [{'fA': 'a1'}, {'fA': 'a2'}]}

    The code snippet shows a query with a regular expression that extracts the values for fieldsA and fieldsB to create the ephemeral fields fld and array.

    parse @message "'fieldsA': '*', 'fieldsB': ['*']" as fld, array

    Named capturing groups

    When you use parse with a regular expression, you can use named capturing groups to capture a pattern into a field. The syntax is parse @message (?<Name>pattern).

    The following example uses a capturing group on a VPC flow log to extract the ENI into a field named NetworkInterface.

    parse @message /(?<NetworkInterface>eni-.*?) / display @timestamp, NetworkInterface
  • unmask

    Use unmask to display all the content of a log event that has some content masked because of a data protection policy. To use this command, you must have the logs:Unmask permission.

    For more information about data protection in log groups, see Help protect sensitive log data with masking.

Note

JSON log events are flattened during ingestion. Currently, parsing nested JSON fields with a glob expression isn't suported. You can only parse JSON log events that include no more than 200 log event fields. When you parse nested JSON fields, you must format the regular expression in your query to match the format of your JSON log event.

Guidelines for working with query commands

You must surround log fields named in queries that include characters other than the @ symbol, period (.), and non-alphanumeric characters in backtick keys (`). For example, the log field foo-bar must be enclosed in backticks (`foo-bar`) because it contains a non-alphanumeric character, the hyphen (-).

Use the display command to show the field or fields that you want to see in your query results. The display command only shows the fields you specify. If your query contains multiple display commands, the query results show only the field or fields that you specified in the final display command.

You can use fields command with the keyword as to create ephemeral fields that use fields and functions in your log events. For example, fields ispresent as isRes creates an ephemeral field named isRes, and the ephemeral field can be used in the rest of your query.

The value of isRes equals 0 or 1, depending on whether resolverArn is a discovered field . If your query contains multiple fields commands and doesn't include a display command, you'll display all of the fields that are specified in the fields commands.

Matches and regular expressions in the filter command

The filter command supports the use of regular expressions. You can use the following comparison operators (=, !=, <, <=, >, >=) and Boolean operators (and, or, and not).

You can use the keyword in to test for set membership and check for elements in an array. To check for elements in an array, put the array after in. You can use the Boolean operator not with in. You can create queries that use in to return log events where fields are string matches. The fields must be complete strings. For example, the following code snippet shows a query that uses in to return log events where the field logGroup is the complete string example_group.

fields @timestamp, @message | filter logGroup in ["example_group"]

You can use the keyword phrases like and not like to match substrings. You can use the regular expression operator =~ to match substrings. To match a substring with like and not like, enclose the substring that you want to match in single or double quotation marks. You can use regular expression patterns with like and not like. To match a substring with the regular expression operator, enclose the substring that you want to match in forward slashes. The following examples contain code snippets that show how you can match substrings using the filter command.

Examples: Match substrings

The following examples return log events where f1 contains the word Exception. All three examples are case sensitive.

The first example matches a substring with like.

fields f1, f2, f3 | filter f1 like "Exception"

The second example matches a substring with like and a regular expression pattern.

fields f1, f2, f3 | filter f1 like /Exception/

The third example matches a substring with a regular expression.

fields f1, f2, f3 | filter f1 =~ /Exception/

Example: Match substrings with wildcards

You can use the period symbol (.) as a wildcard in regular expressions to match substrings. In the following example, the query returns matches where the value for f1 begins with the string ServiceLog.

fields f1, f2, f3 | filter f1 like /ServiceLog./

You can place the asterisk symbol after the period symbol (.*) to create a greedy quantifier that returns as many matches as possible. For example, the following query returns matches where the value for f1 not only begins with the string ServiceLog, but also includes the string ServiceLog.

fields f1, f2, f3 | filter f1 like /ServiceLog.*/

Possible matches can be formatted like the following:

  • ServiceLogSampleApiLogGroup

  • SampleApiLogGroupServiceLog

Example: Exclude substrings from matches

The following example shows a query that returns log events where f1 doesn't contain the word Exception. The example is case senstive.

fields f1, f2, f3 | filter f1 not like "Exception"

Example: Match substrings with case-insensitive patterns

You can match substrings that are case insensitive with like and regular expressions. Place the following parameter (?i) before the substring you want to match. The following example shows a query that returns log events where f1 contains the word Exception or exception.

fields f1, f2, f3 | filter f1 like /(?i)Exception/

Using aliases in queries

Create queries that contain aliases. Use aliases to rename log fields or when extracting values into ephemeral fields. Use the keyword as to give a log field or result an alias. You can use more than one alias in a query. You can use aliases in the following commands:

  • fields

  • parse

  • sort

  • stats

The following examples show how to create queries that contain aliases.

Example

The query contains an alias in the fields command.

fields @timestamp, @message, accountId as ID | sort @timestamp desc | limit 20

The query returns the values for the fields @timestamp, @message, and accountId. The results are sorted in descending order and limited to 20. The values for accountId are listed under the alias ID.

Example

The query contains aliases in the sort and stats commands.

stats count(*) by duration as time | sort time desc

The query counts the number of times the field duration occurs in the log group and sorts the results in descending order. The values for duration are listed under the alias time.

Using comments in queries

CloudWatch Logs Insights supports comments in queries. Use the hash character (#) to set off comments. You can use comments to ignore lines in queries or document queries.

Example: Query

When the following query is run, the second line is ignored.

fields @timestamp, @message, accountId # | filter accountId not like "7983124201998" | sort @timestamp desc | limit 20

Supported operations and functions

CloudWatch Logs Insights supports the following operations and functions.

Arithmetic operators

Arithmetic operators accept numeric data types as arguments and return numeric results. Use arithmetic operators in the filter and fields commands and as arguments for other functions.

Operation Description

a + b

Addition

a - b

Subtraction

a * b

Multiplication

a / b

Division

a ^ b

Exponentiation (2 ^ 3 returns 8)

a % b

Remainder or modulus (10 % 3 returns 1)

Boolean operators

Use the Boolean operators and, or, and not.

Note

Use Boolean operators only in functions that return a value of TRUE or FALSE.

Comparison operators

Comparison operators accept all data types as arguments and return a Boolean result. Use comparison operations in the filter command and as arguments for other functions.

Operator Description

=

Equal

!=

Not equal

<

Less than

>

Greater than

<=

Less than or equal to

>=

Greater than or equal to

Numeric operations

Numeric operations accept numeric data types as arguments and return numeric results. Use numeric operations in the filter and fields commands and as arguments for other functions.

Operation Result type Description

abs(a: number)

number

Absolute value

ceil(a: number)

number

Round to ceiling (the smallest integer that is greater than the value of a)

floor(a: number)

number

Round to floor (the largest integer that is smaller than the value of a)

greatest(a: number, ...numbers: number[])

number

Returns the largest value

least(a: number, ...numbers: number[])

number

Returns the smallest value

log(a: number)

number

Natural log

sqrt(a: number)

number

Square root

Datetime functions

Use datetime functions in the fields and filtercommands and as arguments for other functions. Use these functions to create time buckets for queries with aggregate functions. Use time periods that consist of a number and m for minutes or h for hours. For example, 10m is 10 minutes, and 1h is 1 hour. The following table contains a list of the different datetime functions that you can use in query commands. The table lists each function's result type and contains a description of each function.

Tip

When you create a query command, you can use the time interval selector to select a time period that you want to query. For example, you can set a time period between 5 and 30-minute intervals; 1, 3, and 12-hour intervals; or a custom time frame. You also can set time periods between specific dates.

Function Result type Description

bin(period: Period)

Timestamp

Rounds the value of @timestamp to the given time period and then truncates. For example, bin(5m) rounds the value of @timestamp to the nearest 5 minutes.

You can use this to group multiple log entries together in a query. The following example returns the count of exceptions per hour:

filter @message like /Exception/ | stats count(*) as exceptionCount by bin(1h) | sort exceptionCount desc

The following time units and abbreviations are supported with the bin function. For all units and abbreviations that include more than one character, adding s to pluralize is supported. So both hr and hrs work to specify hours.

  • millisecond ms msec

  • second s sec

  • minute m min

  • hour h hr

  • day d

  • week w

  • month mo mon

  • quarter q qtr

  • year y yr

datefloor(timestamp: Timestamp, period: Period)

Timestamp

Truncates the timestamp to the given period. For example, datefloor(@timestamp, 1h) truncates all values of @timestamp to the bottom of the hour.

dateceil(timestamp: Timestamp, period: Period)

Timestamp

Rounds up the timestamp to the given period and then truncates. For example, dateceil(@timestamp, 1h) truncates all values of @timestamp to the top of the hour.

fromMillis(fieldName: number)

Timestamp

Interprets the input field as the number of milliseconds since the Unix epoch and converts it to a timestamp.

toMillis(fieldName: Timestamp)

number

Converts the timestamp found in the named field into a number representing the milliseconds since the Unix epoch. For example, toMillis(@timestamp) converts the timestamp 2022-01-14T13:18:031.000-08:00 to 1642195111000.

Note

Currently, CloudWatch Logs Insights doesn't support filtering logs with human readable timestamps.

General functions

Use general functions in the fields and filter commands and as arguments for other functions.

Function Result type Description

ispresent(fieldName: LogField)

Boolean

Returns true if the field exists

coalesce(fieldName: LogField, ...fieldNames: LogField[])

LogField

Returns the first non-null value from the list

IP address string functions

Use IP address string functions in the filter and fields commands and as arguments for other functions.

Function Result type Description

isValidIp(fieldName: string)

boolean

Returns true if the field is a valid IPv4 or IPv6 address.

isValidIpV4(fieldName: string)

boolean

Returns true if the field is a valid IPv4 address.

isValidIpV6(fieldName: string)

boolean

Returns true if the field is a valid IPv6 address.

isIpInSubnet(fieldName: string, subnet: string)

boolean

Returns true if the field is a valid IPv4 or IPv6 address within the specified v4 or v6 subnet. When you specify the subnet, use CIDR notation such as 192.0.2.0/24 or 2001:db8::/32, where 192.0.2.0 or 2001:db8:: is the start of the CIDR block.

isIpv4InSubnet(fieldName: string, subnet: string)

boolean

Returns true if the field is a valid IPv4 address within the specified v4 subnet. When you specify the subnet, use CIDR notation such as 192.0.2.0/24 where 192.0.2.0 is the start of the CIDR block..

isIpv6InSubnet(fieldName: string, subnet: string)

boolean

Returns true if the field is a valid IPv6 address within the specified v6 subnet. When you specify the subnet, use CIDR notation such as 2001:db8::/32 where 2001:db8:: is the start of the CIDR block.

Stats aggregation functions

Use aggregation functions in the stats command and as arguments for other functions.

Function Result type Description

avg(fieldName: NumericLogField)

number

The average of the values in the specified field.

count()

count(fieldName: LogField)

number

Counts the log events. count() (or count(*)) counts all events returned by the query, while count(fieldName) counts all records that include the specified field name.

count_distinct(fieldName: LogField)

number

Returns the number of unique values for the field. If the field has very high cardinality (contains many unique values), the value returned by count_distinct is just an approximation.

max(fieldName: LogField)

LogFieldValue

The maximum of the values for this log field in the queried logs.

min(fieldName: LogField)

LogFieldValue

The minimum of the values for this log field in the queried logs.

pct(fieldName: LogFieldValue, percent: number)

LogFieldValue

A percentile indicates the relative standing of a value in a dataset. For example, pct(@duration, 95) returns the @duration value at which 95 percent of the values of @duration are lower than this value, and 5 percent are higher than this value.

stddev(fieldName: NumericLogField)

number

The standard deviation of the values in the specified field.

sum(fieldName: NumericLogField)

number

The sum of the values in the specified field.

Stats non-aggregation functions

Use non-aggregation functions in the stats command and as arguments for other functions.

Function Result type Description

earliest(fieldName: LogField)

LogField

Returns the value of fieldName from the log event that has the earliest timestamp in the queried logs.

latest(fieldName: LogField)

LogField

Returns the value of fieldName from the log event that has the latest timestamp in the queried logs.

sortsFirst(fieldName: LogField)

LogField

Returns the value of fieldName that sorts first in the queried logs.

sortsLast(fieldName: LogField)

LogField

Returns the value of fieldName that sorts last in the queried logs.

String functions

Use string functions in the fields and filter commands and as arguments for other functions.

Function Result type Description

isempty(fieldName: string)

Number

Returns 1 if the field is missing or is an empty string.

isblank(fieldName: string)

Number

Returns 1 if the field is missing, an empty string, or contains only white space.

concat(str: string, ...strings: string[])

string

Concatenates the strings.

ltrim(str: string)

ltrim(str: string, trimChars: string)

string

If the function does not have a second argument, it removes white space from the left of the string. If the function has a second string argument, it does not remove white space. Instead, it removes the characters in trimChars from the left of str. For example, ltrim("xyZxyfooxyZ","xyZ") returns "fooxyZ".

rtrim(str: string)

rtrim(str: string, trimChars: string)

string

If the function does not have a second argument, it removes white space from the right of the string. If the function has a second string argument, it does not remove white space. Instead, it removes the characters of trimChars from the right of str. For example, rtrim("xyZfooxyxyZ","xyZ") returns "xyZfoo".

trim(str: string)

trim(str: string, trimChars: string)

string

If the function does not have a second argument, it removes white space from both ends of the string. If the function has a second string argument, it does not remove white space. Instead, it removes the characters of trimChars from both sides of str. For example, trim("xyZxyfooxyxyZ","xyZ") returns "foo".

strlen(str: string)

number

Returns the length of the string in Unicode code points.

toupper(str: string)

string

Converts the string to uppercase.

tolower(str: string)

string

Converts the string to lowercase.

substr(str: string, startIndex: number)

substr(str: string, startIndex: number, length: number)

string

Returns a substring from the index specified by the number argument to the end of the string. If the function has a second number argument, it contains the length of the substring to be retrieved. For example, substr("xyZfooxyZ",3, 3) returns "foo".

replace(fieldName: string, searchValue: string, replaceValue: string)

string

Replaces all instances of searchValue in fieldName: string with replaceValue.

For example, the function replace(logGroup,"smoke_test","Smoke") searches for log events where the field logGroup contains the string value smoke_test and replaces the value with the string Smoke.

strcontains(str: string, searchValue: string)

number

Returns 1 if str contains searchValue and 0 otherwise.