Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Debugger Built-in Actions for Rules

Use Debugger built-in actions to respond to issues found by Debugger Rule. The Debugger rule_configs class provides tools to configure a list of actions, including automatically stopping training jobs and sending notifications using Amazon Simple Notification Service (Amazon SNS) when the Debugger rules find training issues.

Step 1: Set Up Amazon SNS, Create an SMDebugRules Topic, and Subscribe to the Topic

This section walks you through how to set up an Amazon SNS SMDebugRules topic, subscribe to it, and confirm the subscription to receive notifications from the Debugger rules.

To subscribe to the SMDebugRules topic

After you subscribe to the SMDebugRules topic, you receive the following confirmation message in email or by phone:

For more information about Amazon SNS, see Mobile text messaging (SMS) and Email notifications in the Amazon SNS Developer Guide.

Step 2: Set Up Your IAM Role to Attach Required Policies

In this step, you add the required policies to your IAM role.

For more examples of setting up IAM policies for Amazon SNS, see Example cases for Amazon SNS access control.

Step 3: Configure Debugger Rules with the Built-in Actions

After successfully finishing the required settings in the preceding steps, you can configure the Debugger built-in actions for debugging rules as shown in the following example script. You can choose which built-in actions to use while building the actions list object. The rule_configs is a helper module that provides high-level tools to configure Debugger built-in rules and actions. The following built-in actions are available for Debugger:

You can use all of the built-in actions or a subset of actions by wrapping up using the rule_configs.ActionList() method, which takes the built-in actions and configures a list of actions.

To add all of the three built-in actions to a single rule

If you want to assign all of the three built-in actions to a single rule, configure a Debugger built-in action list while constructing an estimator. Use the following template to construct the estimator, and Debugger will stop training jobs and send notifications through email and text for any rules that you use to monitor your training job progress.

from sagemaker.debugger import Rule, rule_configs

# Configure an action list object for Debugger rules
actions = rule_configs.ActionList(
    rule_configs.StopTraining(), 
    rule_configs.Email("abc@abc.com"), 
    rule_configs.SMS("+1234567890")
)

# Configure rules for debugging with the actions parameter
rules = [
    Rule.sagemaker(
        base_config=rule_configs.built_in_rule(),         # Required
        rule_parameters={"paramter_key": value },        # Optional
        actions=actions
    )
]

estimator = Estimator(
    ...
    rules = rules
)

estimator.fit(wait=False)

To create multiple built-in action objects to assign different actions to a single rule

If you want to assign the built-in actions to be triggered at different threshold values of a single rule, you can create multiple built-in action objects as shown in the following script. To avoid a conflict error by running the same rule, you must submit different rule job names (specify different strings for the rules' name attribute) as shown in the following example script template. This example shows how to set up StalledTrainingRule to take two different actions: send an email to abc@abc.com when a training job stalls for 60 seconds, and stop the training job if stalling for 120 seconds.

from sagemaker.debugger import Rule, rule_configs
import time

base_job_name_prefix= 'smdebug-stalled-demo-' + str(int(time.time()))

# Configure an action object for StopTraining
action_stop_training = rule_configs.ActionList(
    rule_configs.StopTraining()
)

# Configure an action object for Email
action_email = rule_configs.ActionList(
    rule_configs.Email("abc@abc.com")
)

# Configure a rule with the Email built-in action to trigger if a training job stalls for 60 seconds
stalled_training_job_rule_email = Rule.sagemaker(
        base_config=rule_configs.stalled_training_rule(),
        rule_parameters={
                "threshold": "60", 
                "training_job_name_prefix": base_job_name_prefix
        },
        actions=action_email
)
stalled_training_job_rule_text.name="StalledTrainingJobRuleEmail"

# Configure a rule with the StopTraining built-in action to trigger if a training job stalls for 120 seconds
stalled_training_job_rule = Rule.sagemaker(
        base_config=rule_configs.stalled_training_rule(),
        rule_parameters={
                "threshold": "120", 
                "training_job_name_prefix": base_job_name_prefix
        },
        actions=action_stop_training
)
stalled_training_job_rule.name="StalledTrainingJobRuleStopTraining"

estimator = Estimator(
    ...
    rules = [stalled_training_job_rule_email, stalled_training_job_rule]
)

estimator.fit(wait=False)

While the training job is running, the Debugger built-in action sends notification emails and text messages whenever the rule finds issues with your training job. The following screenshot shows an example of email notification for a training job that has a stalled training job issue.

The following screenshot shows an example text notification that Debugger sends when the rule finds a StalledTraining issue.

Considerations for Using the Debugger Built-in Actions