Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Troubleshooting

If your training job fails or behaves unexpectedly, the following sections can help you identify and resolve the issue. Checking your job status can help narrow down whether the issue is in your configuration or your agent, and the agent-specific sections below cover logs and common issues for each deployment path.

Job level debugging

Use the DescribeJob API to check your job's current status and see why it failed. The response includes the job's status, a failure reason if the job has failed, and a timeline of status transitions that shows how far the job progressed before the issue occurred.

aws sagemaker describe-job \
  --job-name "my-agent-rft-job" \
  --job-category AgentRFT \
  --region us-west-2

Key fields to check:

Job CloudWatch logs

Training progress and rollout-level information is logged to the following log group in your account:

/aws/sagemaker/Job/AgentRFT

The log stream name is <job-name>/.

These logs capture training step progress, rollout invocation events, and high-level errors. They can be helpful for understanding how far your job progressed and whether rollouts are being invoked successfully.

If your job fails, check the FailureReason field for details. If it fails during the Training phase, the issue could likely be in your agent. In this case, check your agent logs for more information.

Agent level debugging

Amazon Bedrock AgentCore debugging

If you have deployed your agent to Amazon Bedrock AgentCore, the following can be helpful for investigating agent-side issues.

Agent logs

Your agent container's stdout and stderr output are captured in Amazon CloudWatch Logs in your account. You can find them at the following log group:

/aws/bedrock-agentcore/runtimes/<runtime-name>-<id>-<qualifier>

These logs capture output from your agent code, including errors, stack traces, and SDK messages. These logs can be used for investigating issues related to your agent code, dependencies, or connectivity to the RFT Runtime.

Check agent health

Verify your agent runtime is healthy:

aws bedrock-agentcore-control list-agent-runtimes --region us-west-2

For details on a specific runtime:

aws bedrock-agentcore-control get-agent-runtime \
  --agent-runtime-id <runtime-id> \
  --region us-west-2

Custom agent debugging

If you are using the Lambda forwarder path, issues can occur in the Lambda function itself or in your external agent. The following can be helpful for investigating both.

Lambda forwarder logs

Your Lambda function's execution logs are captured in Amazon CloudWatch Logs. You can find them at the following log group:

/aws/lambda/<function-name>

These logs can be used for investigating issues related to request forwarding, timeouts, or connectivity between the Lambda and your agent. Check for:

Verify connectivity

If your Lambda logs show invocation errors or timeouts, the issue may be that the Lambda cannot reach your agent. The following checks can help confirm whether the connection between your Lambda and agent is working.

Health check — confirm your agent is running:

curl -s "http://$AGENT_ENDPOINT/health"

# Expected: {"status": "ok"}

Lambda test invoke — confirm Lambda can reach your agent:

aws lambda invoke \
  --function-name rft-agent-forwarder \
  --cli-binary-format raw-in-base64-out \
  --payload '{"prompt": "test", "metadata": {"jobArn": "test", "rolloutId": "test-1"}}' \
  --region us-west-2 \
  /tmp/response.json && cat /tmp/response.json

# Note: This will return an InternalServerError because the jobArn "test"
# does not correspond to an active training job. This is expected.
# Success means the Lambda executed and reached your agent — check agent
# logs to confirm the request was received.

If your Lambda executes successfully but the job still fails, your agent logs may have more details. Check your agent logs for errors related to inference calls or reward reporting.

Agent logs

Your agent's own logs depend on where it's deployed. These logs can be used for investigating issues related to your agent code, inference calls to the RFT Runtime, or reward reporting.

For example, if you deployed your agent to Amazon EKS, you can check your agent's logs with:

kubectl logs -l app=external-agent --tail=50

Using CloudTrail for debugging

CloudTrail data events can help confirm whether your agent's calls to the RFT Runtime are succeeding. Look for events with:

If you don't see these events, your agent is not successfully calling the RFT Runtime. Check agent logs and permissions.

Logging API calls with Amazon CloudTrail

Amazon SageMaker AI is integrated with Amazon CloudTrail, a service that provides a record of actions taken by a user, role, or an Amazon service. CloudTrail captures all API calls for Amazon SageMaker AI as events. The calls captured include calls from the Amazon SageMaker AI console and code calls to the Amazon SageMaker AI API operations. Using the information collected by CloudTrail, you can determine the request that was made to Amazon SageMaker AI, the IP address from which the request was made, when it was made, and additional details.

Every event or log entry contains information about who generated the request. The identity information helps you determine the following:

CloudTrail is active in your Amazon account when you create the account and you automatically have access to the CloudTrail Event history. The CloudTrail Event history provides a viewable, searchable, downloadable, and immutable record of the past 90 days of recorded management events in an Amazon Region. For more information, see Working with CloudTrail Event history in the Amazon CloudTrail User Guide. There are no CloudTrail charges for viewing the Event history.

For an ongoing record of events in your Amazon account past 90 days, create a trail or a CloudTrail Lake event data store.

CloudTrail trails

A trail enables CloudTrail to deliver log files to an Amazon S3 bucket. All trails created using the Amazon Management Console are multi-Region. You can create a single-Region or a multi-Region trail by using the Amazon CLI. Creating a multi-Region trail is recommended because you capture activity in all Amazon Regions in your account. If you create a single-Region trail, you can view only the events logged in the trail's Amazon Region. For more information about trails, see Creating a trail for your Amazon account and Creating a trail for an organization in the Amazon CloudTrail User Guide.

You can deliver one copy of your ongoing management events to your Amazon S3 bucket at no charge from CloudTrail by creating a trail, however, there are Amazon S3 storage charges. For more information about CloudTrail pricing, see Amazon CloudTrail Pricing. For information about Amazon S3 pricing, see Amazon S3 Pricing.

CloudTrail Lake event data stores

CloudTrail Lake lets you run SQL-based queries on your events. CloudTrail Lake converts existing events in row-based JSON format to Apache ORC format. ORC is a columnar storage format that is optimized for fast retrieval of data. Events are aggregated into event data stores, which are immutable collections of events based on criteria that you select by applying advanced event selectors. The selectors that you apply to an event data store control which events persist and are available for you to query. For more information about CloudTrail Lake, see Working with Amazon CloudTrail Lake in the Amazon CloudTrail User Guide.

CloudTrail Lake event data stores and queries incur costs. When you create an event data store, you choose the pricing option you want to use for the event data store. The pricing option determines the cost for ingesting and storing events, and the default and maximum retention period for the event data store. For more information about CloudTrail pricing, see Amazon CloudTrail Pricing.

SageMaker AI data events in CloudTrail

Data events provide information about the resource operations performed on or in a resource (for example, reading or writing to an Amazon S3 object). These are also known as data plane operations. Data events are often high-volume activities. By default, CloudTrail doesn't log data events. The CloudTrail Event history doesn't record data events.

Additional charges apply for data events. For more information about CloudTrail pricing, see Amazon CloudTrail Pricing.

You can log data events for various Amazon SageMaker AI resource types by using the CloudTrail console, Amazon CLI, or CloudTrail API operations. For more information about how to log data events, see Logging data events with the Amazon Management Console and Logging data events with the Amazon Command Line Interface in the Amazon CloudTrail User Guide.

The following table lists the Amazon SageMaker AI resource types for which you can log data events:

You can configure advanced event selectors to filter on the eventName, readOnly, and resources.ARN fields to log only those events that are important to you. For more information about these fields, see AdvancedFieldSelector in the Amazon CloudTrail API Reference.

Example: Log data events for a SageMaker endpoint and job

The following example shows how to use the put-event-selectors Amazon CLI command to add advanced event selectors:

[
  {
    "FieldSelectors": [
      { "Field": "eventCategory", "Equals": ["Data"] },
      { "Field": "resources.ARN", "Equals": ["arn:aws:sagemaker:us-east-1:111122223333:endpoint/your-inference-endpoint-arn"] },
      { "Field": "resources.type", "Equals": ["AWS::SageMaker::Endpoint"] }
    ]
  },
  {
    "FieldSelectors": [
      { "Field": "eventCategory", "Equals": ["Data"] },
      { "Field": "resources.ARN", "Equals": ["arn:aws:sagemaker:us-east-1:111122223333:job/your-job-arn"] },
      { "Field": "resources.type", "Equals": ["AWS::SageMaker::Job"] }
    ]
  }
]

Then run:

aws cloudtrail put-event-selectors \
    --trail-name your-trail-name \
    --advanced-event-selectors=file://advanced-event-selectors.json

SageMaker AI management events in CloudTrail

Management events provide information about management operations that are performed on resources in your Amazon account. These are also known as control plane operations. By default, CloudTrail logs management events.

Amazon SageMaker AI logs all Amazon SageMaker AI control plane operations as management events. For a list of the Amazon SageMaker AI control plane operations that Amazon SageMaker AI logs to CloudTrail, see the Amazon SageMaker AI API Reference.

CloudTrail event examples

For information about CloudTrail record contents, see CloudTrail record contents in the Amazon CloudTrail User Guide.

Model packages and checkpoints

Overview

During multi-turn RL training, the platform periodically saves the model's learned parameters as checkpoints. These checkpoints are stored as SageMaker Model Packages within Model Package Groups, enabling versioning, lineage tracking, and cross-job continuity.

Key concepts

Model Package

A Model Package is a versioned, immutable artifact in SageMaker AI that contains trained model weights at a specific point in time. Each checkpoint produced during training is stored as a Model Package. A Model Package has:

Model Package Group

A Model Package Group is a container that holds multiple Model Package versions. Multi-turn RL uses two separate groups:

You specify both when creating a job:

{
  "ModelPackageConfig": {
    "OutputModelPackageGroupArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package-group/my-final-models",
    "IntermediateCheckpointModelPackageGroupArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package-group/my-intermediate-checkpoints"
  }
}

Checkpoint types

Resumable Checkpoint (Full State)

Model Checkpoint (Weights Only)

Resuming interrupted training

If a training job fails or is stopped mid-training, you can start a new job that resumes from the exact point where the previous job left off. The platform loads the full training state (weights + optimizer + step counter) from a resumable checkpoint.

To resume, specify a resumable checkpoint (from the Intermediate Checkpoint Model Package Group) as InputModelPackageArn:

{
  "ModelPackageConfig": {
    "OutputModelPackageGroupArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package-group/my-final-models",
    "IntermediateCheckpointModelPackageGroupArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package-group/my-intermediate-checkpoints",
    "InputModelPackageArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package/my-intermediate-checkpoints/5"
  }
}

Requirements:

Iterative training (continued training)

Iterative training lets you build on a previously trained model with new hyperparameters, a different dataset, or a different training configuration. Unlike resuming, this starts a fresh training run that initializes from the trained LoRA weights but with a fresh optimizer state.

To do iterative training, specify a model checkpoint (from the Output Model Package Group) as InputModelPackageArn:

{
  "ModelPackageConfig": {
    "OutputModelPackageGroupArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package-group/my-final-models",
    "IntermediateCheckpointModelPackageGroupArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package-group/my-intermediate-checkpoints",
    "InputModelPackageArn": "arn:aws:sagemaker:us-west-2:123456789012:model-package/my-final-models/3"
  }
}

What you can change between iterations:

What must stay the same:

Typical use cases:

Checkpoint lifecycle

Training Step 1 → Intermediate Checkpoint (Resumable)
Training Step 1 → Intermediate Checkpoint (HFCompatible)
...
Training Step N-1 → Intermediate Checkpoint (Resumable)
Training Step N-1 → Intermediate Checkpoint (HFCompatible)
...
Training Step N (final) → Model Checkpoint (HuggingFace LoRA) → Output Model Package Group

When a job completes successfully: The final model weights are saved as a Model Package in the Output Model Package Group. The OutputModelPackageArn field on the job record contains the ARN of the final model.

When a job fails or is stopped: The last intermediate checkpoint is promoted to the Output Model Package Group (best-effort).

Best practices for checkpoints