Debug lifecycle configurations
The following topics show how to get information about and debug your lifecycle configurations.
Verify lifecycle configuration process from CloudWatch Logs
Lifecycle configurations only log STDOUT and STDERR.
STDOUT is the default output for bash scripts. You can write to
STDERR by appending >&2 to the end of a bash command.
For example, echo 'hello'>&2.
Logs for your lifecycle configurations are published to your Amazon Web Services account using
Amazon CloudWatch. These logs can be found in the /aws/sagemaker/studio log stream
in the CloudWatch console.
Open the CloudWatch console at https://console.amazonaws.cn/cloudwatch/
. -
Choose Logs from the left navigation pane. From the dropdown menu, select Log groups.
-
On the Log groups page, search for
aws/sagemaker/studio. -
Select the log group.
-
On the Log group details page, choose the Log streams tab.
-
To find the logs for a specific app, search the log streams using the following format:
domain-id/user-profile-name/app-type/app-nameThe following search string finds the lifecycle configuration logs for the domain
d-m85lcu8vbqmz, user profilei-sonic-js, application typeJupyterLab, and application nametest-lcc-echo:d-m85lcu8vbqmz/i-sonic-js/JupyterLab/test-lcc-echo -
To view the script execution logs, select the log stream appended with
LifecycleConfigOnStart.
Lifecycle configuration timeout
There is a lifecycle configuration timeout limitation of 5 minutes. If a lifecycle configuration script takes longer than 5 minutes to run, you get an error.
To resolve this error, make sure that your lifecycle configuration script completes in less than 5 minutes.
To help decrease the runtime of scripts, try the following:
-
Reduce unnecessary steps. For example, limit which conda environments to install large packages in.
-
Run tasks in parallel processes.
-
Use the nohup command in your script to make sure that hangup signals are ignored so that the script runs without stopping.