CloudWatch Logs Insights query syntax
With CloudWatch Logs Insights, you use a query language to query your log groups. The query syntax supports different functions and operations that include but aren't limited to general functions, arithmetic and comparison operations, and regular expressions. Create queries that contain multiple commands. Separate commands with the pipe character (|). Create queries that contain comments. Set off comments with the hash character (#).
Note
CloudWatch Logs Insights automatically discovers fields
for different log types
and generates fields
that start with the @ character.
For more information
about these fields,
see Supported logs and discovered fields
CloudWatch Logs Insights query commands
This section includes a list of the supported CloudWatch Logs Insights query commands.
Note
Not all of the example queries in this section will work on your log events. The example queries in this section are meant to show how you can format your queries using the supported CloudWatch Logs Insights query commands. You may need to format the queries to match the format of your log events. For more query examples, see Sample queries.
-
display
Use
display
to show a specific field or fields in query results.Example: Display one field
The code snippet shows an example of a query that uses the parse command to extract data from
@message
to create the ephemeral fieldsloggingType
andloggingMessage
. The query returns all log events where the values forloggingType
are ERROR.display
shows only the values forloggingMessage
in the query results.fields @message | parse @message "[*] *" as loggingType, loggingMessage | filter loggingType = "ERROR" | display loggingMessage
Tip
Use
display
only once in a query. If you usedisplay
more than once in a query, the query results show the field specified in the last occurrence ofdisplay
command being used. -
fields
Use
fields
to show specific fields in query results.Example: Display specific fields
The code snippet shows an example of a query that returns 20 log events and displays them in descending order. The values for
@timestamp
and@message
are shown in the query results.fields @timestamp, @message | sort @timestamp desc | limit 20
Tip
Use this command when you don't want to use
display
.fields
supports different functions and operations for modifying field values and creating new fields that can be used in queries. -
filter
Use
filter
to get log events that match one or more conditions.Example: Filter log events using one condition
The code snippet shows an example of a query that returns all log events where the value for
range
is greater than 3000. The query limits the results to 20 log events and sorts the logs events by@timestamp
and in descending order.fields @timestamp, @message | filter (range>3000) | sort @timestamp desc | limit 20
Example: Filter log events using more than one condition
You can use the keywords
and
andor
to combine more than one condition.The code snippet shows an example of a query that returns log events where the value for
range
is greater than 3000 and value foraccountId
is equal to 123456789012. The query limits the results to 20 log events and sorts the logs events by@timestamp
and in descending order.fields @timestamp, @message | filter (range>3000 and accountId=123456789012) | sort @timestamp desc | limit 20
-
stats
Use
stats
to calculate aggregate statistics with log field values. -
sort
Use
sort
to display log events in ascending (asc
) or descending (desc
) order. -
limit
Use
limit
to specify the number of log events that you want your query to return. -
parse
Use
parse
to extract data from a log field and create an ephemeral field that you can process in your query.parse
supports both glob mode using wildcards, and regular expressions.You can parse nested JSON fields with a regular expression.
Example: Parsing a nested JSON field
The code snippet shows how to parse a JSON log event that's been flattened during ingestion.
{'fieldsA': 'logs', 'fieldsB': [{'fA': 'a1'}, {'fA': 'a2'}]}
The code snippet shows a query with a regular expression that extracts the values for
fieldsA
andfieldsB
to create the ephemeral fieldsfld
andarray
.parse @message "'fieldsA': '*', 'fieldsB': ['*']" as fld, array
Named capturing groups
When you use
parse
with a regular expression, you can use named capturing groups to capture a pattern into a field. The syntax isparse @message (?<Name>pattern).
The following example uses a capturing group on a VPC flow log to extract the ENI into a field named
NetworkInterface
.parse @message /(?<NetworkInterface>eni-.*?) / display @timestamp, NetworkInterface
-
unmask
Use
unmask
to display all the content of a log event that has some content masked because of a data protection policy. To use this command, you must have thelogs:Unmask
permission.For more information about data protection in log groups, see Help protect sensitive log data with masking.
Note
JSON log events are flattened during ingestion. Currently, parsing nested JSON fields with a glob expression isn't suported. You can only parse JSON log events that include no more than 200 log event fields. When you parse nested JSON fields, you must format the regular expression in your query to match the format of your JSON log event.
Guidelines for working with query commands
You must surround log fields
named in queries
that include characters
other than the @
symbol,
period (.
),
and non-alphanumeric characters
in backtick keys (`
).
For example,
the log field foo-bar
must be enclosed
in backticks (`foo-bar`
)
because it contains a non-alphanumeric character,
the hyphen (-
).
Use the display
command
to show the field or fields
that you want to see
in your query results.
The display
command
only shows the fields
you specify.
If your query contains multiple display
commands,
the query results show only the field or fields
that you specified in the final display
command.
You can use fields
command
with the keyword as
to create ephemeral fields
that use fields and functions
in your log events.
For example,
fields ispresent as isRes
creates an ephemeral field
named isRes
,
and the ephemeral field can be used
in the rest of your query.
The value of isRes
equals 0 or 1,
depending on whether resolverArn
is a discovered field
.
If your query contains multiple fields
commands
and doesn't include a display
command,
you'll display all of the fields
that are specified
in the fields
commands.
Matches and regular expressions in the filter command
The filter command supports the
use of regular expressions.
You can use the following comparison operators (=
, !=
, <
, <=
, >
, >=
)
and Boolean operators (and
, or
, and not
).
You can use the keyword in
to test
for set membership
and check
for elements
in an array.
To check
for elements
in an array,
put the array
after in
.
You can use the Boolean operator not
with in
.
You can create queries
that use in
to return log events
where fields
are string matches.
The fields must be complete strings.
For example,
the following code snippet shows
a query
that uses in
to return log events
where the field logGroup
is the complete string
example_group
.
fields @timestamp, @message | filter logGroup in ["example_group"]
You can use the keyword phrases like
and not like
to match substrings.
You can use the regular expression operator =~
to match substrings.
To match a substring
with like
and not like
,
enclose the substring
that you want to match
in single or double quotation marks.
You can use regular expression patterns
with like
and not like
.
To match a substring
with the regular expression operator,
enclose the substring
that you want to match
in forward slashes.
The following examples contain code snippets
that show how you can match substrings
using the filter
command.
Examples: Match substrings
The following examples return log events
where f1
contains the word Exception.
All three examples are case sensitive.
The first example matches a substring
with like
.
fields f1, f2, f3 | filter f1 like "Exception"
The second example matches a substring
with like
and a regular expression pattern.
fields f1, f2, f3 | filter f1 like /Exception/
The third example matches a substring with a regular expression.
fields f1, f2, f3 | filter f1 =~ /Exception/
Example: Match substrings with wildcards
You can use the period symbol (.
)
as a wildcard
in regular expressions
to match substrings.
In the following example,
the query returns matches
where the value
for f1
begins with the string ServiceLog
.
fields f1, f2, f3 | filter f1 like /ServiceLog./
You can place the asterisk symbol
after the period symbol (.*
)
to create a greedy quantifier
that returns
as many matches
as possible.
For example,
the following query
returns matches
where the value
for f1
not only begins
with the string ServiceLog
,
but also includes the string ServiceLog
.
fields f1, f2, f3 | filter f1 like /ServiceLog.*/
Possible matches can be formatted like the following:
-
ServiceLogSampleApiLogGroup
-
SampleApiLogGroupServiceLog
Example: Exclude substrings from matches
The following example shows a query
that returns log events
where f1
doesn't contain the word Exception.
The example is case senstive.
fields f1, f2, f3 | filter f1 not like "Exception"
Example: Match substrings with case-insensitive patterns
You can match substrings
that are case insensitive
with like
and regular expressions.
Place the following parameter (?i)
before the substring you want
to match.
The following example shows a query
that returns log events
where f1
contains the word Exception
or exception.
fields f1, f2, f3 | filter f1 like /(?i)Exception/
Using aliases in queries
Create queries
that contain aliases.
Use aliases
to rename log fields or
when extracting values
into ephemeral fields.
Use the keyword as
to give a log field or result an alias.
You can use more than one alias
in a query.
You can use aliases
in the following commands:
-
fields
-
parse
-
sort
-
stats
The following examples show how to create queries that contain aliases.
Example
The query contains an alias
in the fields
command.
fields @timestamp, @message, accountId as ID | sort @timestamp desc | limit 20
The query returns the values
for the fields @timestamp
, @message
, and accountId
.
The results are sorted
in descending order and limited
to 20.
The values
for accountId
are listed
under the alias ID
.
Example
The query contains aliases
in the sort
and stats
commands.
stats count(*) by duration as time | sort time desc
The query counts the number
of times
the field duration
occurs
in the log group
and sorts the results
in descending order.
The values
for duration
are listed
under the alias time
.
Using comments in queries
CloudWatch Logs Insights supports comments in queries. Use the hash character (#) to set off comments. You can use comments to ignore lines in queries or document queries.
Example: Query
When the following query is run, the second line is ignored.
fields @timestamp, @message, accountId # | filter accountId not like "7983124201998" | sort @timestamp desc | limit 20
Supported operations and functions
CloudWatch Logs Insights supports the following operations and functions.
Arithmetic operators
Arithmetic operators accept numeric data types
as arguments
and return numeric results.
Use arithmetic operators
in the filter
and fields
commands
and
as arguments for other functions.
Operation | Description |
---|---|
|
Addition |
|
Subtraction |
|
Multiplication |
|
Division |
|
Exponentiation
( |
|
Remainder or modulus
( |
Boolean operators
Use the Boolean operators and
, or
, and not
.
Note
Use Boolean operators only in functions that return a value of TRUE or FALSE.
Comparison operators
Comparison operators accept all data types
as arguments
and return a Boolean result.
Use comparison operations
in the filter
command
and
as arguments for other functions.
Operator | Description |
---|---|
|
Equal |
|
Not equal |
|
Less than |
|
Greater than |
|
Less than or equal to |
|
Greater than or equal to |
Numeric operations
Numeric operations accept numeric data types
as arguments
and return numeric results.
Use numeric operations in the filter
and fields
commands
and
as arguments for other functions.
Operation | Result type | Description |
---|---|---|
|
number |
Absolute value |
|
number |
Round
to ceiling (the smallest integer
that is
greater than the value
of |
|
number |
Round
to floor (the largest integer
that is
smaller than the value
of |
|
number |
Returns the largest value |
|
number |
Returns the smallest value |
|
number |
Natural log |
|
number |
Square root |
Datetime functions
Use datetime functions
in the fields
and filter
commands
and
as arguments
for other functions.
Use these functions
to create time buckets
for queries
with aggregate functions.
Use time periods
that consist
of a number
and m
for minutes
or h
for hours.
For example,
10m
is 10 minutes,
and 1h
is 1 hour.
The following table contains a list
of the different datetime functions
that you can use
in query commands.
The table lists each function's result type
and contains a description
of each function.
Tip
When you create a query command, you can use the time interval selector to select a time period that you want to query. For example, you can set a time period between 5 and 30-minute intervals; 1, 3, and 12-hour intervals; or a custom time frame. You also can set time periods between specific dates.
Function | Result type | Description |
---|---|---|
|
Timestamp |
Rounds the value of You can use this to group multiple log entries together in a query. The following example returns the count of exceptions per hour:
The following time units and abbreviations are supported with the
|
|
Timestamp |
Truncates the timestamp to the given period. For example, |
|
Timestamp |
Rounds up the timestamp to the given period and then truncates. For example, |
|
Timestamp |
Interprets the input field as the number of milliseconds since the Unix epoch and converts it to a timestamp. |
|
number |
Converts the timestamp
found
in the named field
into a number representing the milliseconds since the Unix epoch.
For example,
|
Note
Currently, CloudWatch Logs Insights doesn't support filtering logs with human readable timestamps.
General functions
Use general functions
in the fields
and filter
commands
and
as arguments
for other functions.
Function | Result type | Description |
---|---|---|
|
Boolean |
Returns |
|
LogField |
Returns the first non-null value from the list |
IP address string functions
Use IP address string functions
in the filter
and fields
commands
and
as arguments
for other functions.
Function | Result type | Description |
---|---|---|
|
boolean |
Returns |
|
boolean |
Returns |
|
boolean |
Returns |
|
boolean |
Returns |
|
boolean |
Returns |
|
boolean |
Returns |
Stats aggregation functions
Use aggregation functions
in the stats
command
and
as arguments
for other functions.
Function | Result type | Description |
---|---|---|
|
number |
The average of the values in the specified field. |
|
number |
Counts the log events. |
|
number |
Returns the number of unique values for the field. If the field has very high cardinality
(contains many unique values), the value returned by |
|
LogFieldValue |
The maximum of the values for this log field in the queried logs. |
|
LogFieldValue |
The minimum of the values for this log field in the queried logs. |
|
LogFieldValue |
A percentile indicates the relative standing of
a value in a dataset. For example,
|
|
number |
The standard deviation of the values in the specified field. |
|
number |
The sum of the values in the specified field. |
Stats non-aggregation functions
Use non-aggregation functions
in the stats
command
and
as arguments
for other functions.
Function | Result type | Description |
---|---|---|
|
LogField |
Returns the value of |
|
LogField |
Returns the value of |
|
LogField |
Returns the value of |
|
LogField |
Returns the value of |
String functions
Use string functions
in the fields
and filter
commands
and
as arguments
for other functions.
Function | Result type | Description |
---|---|---|
|
Number |
Returns |
|
Number |
Returns |
|
string |
Concatenates the strings. |
|
string |
If the function does not have a second argument, it removes white space from the left of the string. If the
function has a second string argument, it does not remove white space. Instead, it removes the characters
in |
|
string |
If the function does not have a second argument, it removes white space from the right of the string. If the
function has a second string argument, it does not remove white space. Instead, it removes the characters
of |
|
string |
If the function does not have a second argument, it removes white space from both ends of the string. If the
function has a second string argument, it does not remove white space. Instead, it removes the characters
of |
|
number |
Returns the length of the string in Unicode code points. |
|
string |
Converts the string to uppercase. |
|
string |
Converts the string to lowercase. |
|
string |
Returns a substring from the index specified by the number argument to the end of
the string. If the function has a second number argument, it contains the length of the
substring to be retrieved. For example, |
|
string |
Replaces all instances of For example,
the function |
|
number |
Returns 1 if |