Updating Debugger system monitoring and framework profiling configuration while a training job is running - Amazon SageMaker AI
Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Updating Debugger system monitoring and framework profiling configuration while a training job is running

If you want to activate or update the Debugger monitoring configuration for a training job that is currently running, use the following SageMaker AI estimator extension methods:

  • To activate Debugger system monitoring for a running training job and receive a Debugger profiling report, use the following:

    estimator.enable_default_profiling()

    When you use the enable_default_profiling method, Debugger initiates the default system monitoring and the ProfileReport built-in rule, which generates a comprehensive profiling report at the end of the training job. This method can be called only if the current training job is running without both Debugger monitoring and profiling.

    For more information, see estimator.enable_default_profiling in the Amazon SageMaker Python SDK.

  • To update system monitoring configuration, use the following:

    estimator.update_profiler(
    system_monitor_interval_millis=500
)

    For more information, see estimator.update_profiler in the Amazon SageMaker Python SDK.