Amazon WAF Bot Control rule group
VendorName: AWS
, Name:
, WCU:
50AWSManagedRulesBotControlRuleSet
The Bot Control managed rule group provides rules that manage requests from bots. Bots can consume excess resources, skew business metrics, cause downtime, and perform malicious activities.
Protection levels
The Bot Control managed rule group provides two levels of protection that you can choose from:
-
Common – Detects a variety of self-identifying bots, such as web scraping frameworks, search engines, and automated browsers. Bot Control protections at this level identify common bots using traditional bot detection techniques, such as static request data analysis. The rules label traffic from these bots and block the ones that they cannot verify.
-
Targeted – Includes the common-level protections and adds targeted detection for sophisticated bots that do not self identify. Targeted protections mitigate bot activity using a combination of rate limiting and CAPTCHA and background browser challenges.
TGT_
– Rules that provide targeted protection have names that begin withTGT_
. All targeted protections use detection techniques such as browser interrogation, fingerprinting, and behavior heuristics to identify bad bot traffic.TGT_ML_
– Targeted protection rules that use machine learning have names that begin withTGT_ML_
. These rules use automated, machine-learning analysis of website traffic statistics to detect anomalous behavior indicative of distributed, coordinated bot activity. Amazon WAF analyzes statistics about your website traffic such as timestamps, browser characteristics, and previous URL visited, to improve the Bot Control machine learning model. Machine learning capabilities are enabled by default, but you can disable them in your rule group configuration. When machine learning is disabled, Amazon WAF does not evaluate these rules.
The targeted protection level and the Amazon WAF rate-based rule statement both provide rate limiting. For a comparison of the two options, see Options for rate limiting in rate-based rules and targeted Bot Control rules.
Using this rule group
This rule group is part of the intelligent threat mitigation protections in Amazon WAF. For information, see Amazon WAF intelligent threat mitigation.
Note
You are charged additional fees when you use this managed rule group. For more information, see Amazon WAF Pricing
To keep your costs down and to be sure you're managing your web traffic as you want, use this rule group in accordance with the guidance at Best practices for intelligent threat mitigation.
The Bot Control rule group doesn't provide SNS update notifications.
Token labels
This rule group uses Amazon WAF token management to inspect and label web requests according to the status of their Amazon WAF tokens. Amazon WAF uses tokens for client session tracking and verification.
Amazon WAF applies one of the following labels when it inspects a web request's token and challenge timestamp. Amazon WAF doesn't add labeling about the status of the CAPTCHA timestamp.
-
awswaf:managed:token:accepted
– The request token is present and has an unexpired challenge timestamp. -
awswaf:managed:token:rejected
– The request token is present but is either corrupt or has an expired challenge timestamp. -
awswaf:managed:token:absent
– The request doesn't have a token.
For more information, see Amazon WAF web request tokens.
Bot Control labels
The Bot Control managed rule group generates labels with the namespace prefix
awswaf:managed:aws:bot-control:
followed by the custom
namespace and label name. The rule group might add more than one label
to a request.
Each label reflects the Bot Control rule findings:
-
awswaf:managed:aws:bot-control:bot:
– Information about the bot associated with the request.-
awswaf:managed:aws:bot-control:bot:name:
– The bot name, if one is available, for example, the custom namespaces<name>
bot:name:slurp
,bot:name:googlebot
, andbot:name:pocket_parser
. -
awswaf:managed:aws:bot-control:bot:category:
– The category of bot, as defined by Amazon WAF, for example,<category>
bot:category:search_engine
andbot:category:content_fetcher
. -
awswaf:managed:aws:bot-control:bot:organization:
– The bot's publisher, for example,<organization>
bot:organization:google
. -
awswaf:managed:aws:bot-control:bot:verified
– Used to indicate a bot that identifies itself and that Bot Control has been able to verify. This is used for common desirable bots, and can be useful when combined with category labels likebot:category:search_engine
or name labels likebot:name:googlebot
.Note
Bot Control uses the IP address from the web request origin to help determine whether a bot is verified. You can’t configure it to use the Amazon WAF forwarded IP configuration, to inspect a different IP address source. If you have verified bots that route through a proxy or load balancer, you can add a rule that runs before the Bot Control rule group to help with this. Configure your new rule to use the forwarded IP address and explicitly allow requests from the verified bots. For information about using forwarded IP addresses, see Forwarded IP address.
-
awswaf:managed:aws:bot-control:bot:user_triggered:verified
– Used to indicate a bot that is similar to a verified bot, but that might be directly invoked by end users. This category of bot is treated by the Bot Control rules like an unverified bot. -
awswaf:managed:aws:bot-control:bot:developer_platform:verified
– Used to indicate a bot that is similar to a verified bot, but that is used by developer platforms for scripting, for example Google Apps Script. This category of bot is treated by the Bot Control rules like an unverified bot. -
awswaf:managed:aws:bot-control:bot:unverified
– Used to indicate a bot that identifies itself, so it can be named and categorized, but that doesn't publish information that can be used to independently verify its identify. These types of bot signatures can be falsified, and so are treated as unverified.
-
-
awswaf:managed:aws:bot-control:signal:
– Used for attributes of the request that are indicative of bots that are not more commonly used or verified.<signal-details>
-
awswaf:managed:aws:bot-control:targeted:
– Used by the Bot Control targeted protections.<additional-details>
You can retrieve all labels for a rule group through the API by calling
DescribeManagedRuleGroup
. The labels are listed in the
AvailableLabels
property in the response.
The Bot Control managed rule group applies labels to a set of verifiable bots that are commonly
allowed. The rule group doesn't block these verified bots and doesn't apply
any signal:
labels. If you want, you can block them, or a
subset of them by writing a custom rule that uses the labels applied by the
Bot Control managed rule group. For more information about this and examples, see
Amazon WAF Bot Control.
Bot Control rules listing
The following table lists the Bot Control rules.
Note
The information that we publish for the rules in the Amazon Managed Rules rule groups is intended to provide you
with enough information to use the rules while not providing information that
bad actors could use to circumvent the rules. If you need more information than
you find in this documentation, contact the Amazon Web Services Support Center
Rule name | Description |
---|---|
CategoryAdvertising |
Inspects for bots that are used for advertising purposes. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryArchiver |
Inspects for bots that are used for archiving purposes. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryContentFetcher |
Inspects for bots that are fetching content on behalf of a user. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryEmailClient |
Inspects for email clients. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryHttpLibrary |
Inspects for HTTP libraries that are used by bots. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryLinkChecker |
Inspects for bots that check for broken links. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryMiscellaneous |
Inspects for miscellaneous bots. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryMonitoring |
Inspects for bots that are used for monitoring purposes. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryScrapingFramework |
Inspects for web scraping frameworks. Rule action, applied only to unverified bots: Block Label: For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategorySearchEngine |
Inspects for search engine bots. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategorySecurity |
Inspects for security-related bots. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategorySeo |
Inspects for bots that are used for search engine optimization. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategorySocialMedia |
Inspects for bots that are used by social media platforms to provide content summaries. Rule action, applied only to unverified bots: Block Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
CategoryAI |
Inspects for artificial intelligence (AI) bots. Rule action: Block Label:
|
SignalAutomatedBrowser |
Inspects the request's token for indicators that the client browser might be automated. For more information, see Token characteristics. Rule action: Block Label:
|
SignalKnownBotDataCenter |
Inspects for data centers that are typically used by bots. Rule action: Block Label:
|
SignalNonBrowserUserAgent |
Inspects for user agent strings that don't seem to be from a web browser. Rule action: Block Label:
|
TGT_VolumetricIpTokenAbsent |
Inspects for 5 or more requests from a client in the last 5 minutes that don't include a valid challenge token. For information about tokens, see Amazon WAF web request tokens. NoteThe threshold that this rule applies can vary slightly due to latency. This rule handles missing tokens differently from the
token labeling: Rule action, applied only to clients that are not verified bots: Challenge Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
This rule is different from the token labeling:
|
TGT_VolumetricSession |
Inspects for an abnormally high number of requests from a client session in any 5 minute window. The evaluation is based on a comparison to standard volumetric baselines that Amazon WAF maintains using historic traffic patterns. This inspection only applies when the web request has a token. Tokens are added to requests by the application integration SDKs and by the rule actions CAPTCHA and Challenge. For more information, see Amazon WAF web request tokens. NoteThis rule can take 5 minutes to go into effect after you enable it. Bot Control identifies anomalous behavior in your web traffic by comparing the current traffic to traffic baselines that Amazon WAF computes. Rule action, applied only to clients that are not verified bots: CAPTCHA Label:
The rule group applies the following labels to
medium volume and lower volume requests that are
above a minimum threshold. For these levels, the
rule takes no action, regardless of whether the
client is verified:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
TGT_SignalAutomatedBrowser |
Inspects the request's token for indicators that the client browser might be automated. For more information, see Token characteristics. This inspection only applies when the web request has a token. Tokens are added to requests by the application integration SDKs and by the rule actions CAPTCHA and Challenge. For more information, see Amazon WAF web request tokens. Rule action, applied only to clients that are not verified bots: CAPTCHA Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
TGT_SignalBrowserInconsistency |
Inspects for inconsistent browser interrogation data. For more information, see Token characteristics. This inspection only applies when the web request has a token. Tokens are added to requests by the application integration SDKs and by the rule actions CAPTCHA and Challenge. For more information, see Amazon WAF web request tokens. Rule action, applied only to clients that are not verified bots: CAPTCHA Label:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|
TGT_TokenReuseIp |
Inspects for the use of a single token among more than 5 distinct IP addresses. NoteThe thresholds that this rule applies can vary slightly due to latency. A few requests might make it through beyond the limit before the rule action is applied. Rule action: Count Label:
|
TGT_ML_CoordinatedActivityMedium
and
TGT_ML_CoordinatedActivityHigh |
Inspect for anomalous behavior consistent with distributed, coordinated bot activity. The rule levels indicate the level of confidence that a group of requests are participants in a coordinated attack. NoteThese rules only run if the rule group is configured to use machine learning (ML). For information about configuring this choice, see Adding the Amazon WAF Bot Control managed rule group to your web ACL. Amazon WAF performs this inspection through machine learning analysis of website traffic statistics. Amazon WAF analyzes web traffic every few minutes and optimizes the analysis for the detection of low intensity, long-duration bots that are distributed across many IP addresses. These rules might match on a very small number of requests before determining that a coordinated attack is not underway. So if you see just a match or two, the results might be false positives. If you see a lot of matches coming out of these rules however, then you're probably experiencing a coordinated attack. NoteThese rules can take up to 24 hours to go into effect after you enable the Bot Control targeted rules with the ML option. Bot Control identifies anomalous behavior in your web traffic by comparing the current traffic to traffic baselines that Amazon WAF has computed. Amazon WAF only computes the baselines while you're using the Bot Control targeted rules with the ML option, and it can take up to 24 hours to establish meaningful baselines. Rule actions, applied only to clients that are not verified bots:
Labels:
For verified bots, the rule group takes no action, but it adds
the rule labeling plus the label
|