Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Understanding results and output

Batch evaluation results come in two layers: aggregate summaries in the API response, and per-session detail in CloudWatch Logs.

Aggregate results

When a batch evaluation completes, the GetBatchEvaluation response includes an evaluationResults object with aggregate summaries.

Session counts

Per-evaluator summaries

Each entry in evaluatorSummaries provides aggregate metrics for one evaluator, or for one dimension of a multi-dimension evaluator:

A multi-dimension evaluator has one entry per dimension. A separate entry for the evaluator, without evaluationDimensionName, counts the sessions that fail before the judge model runs.

Example response

{
    "batchEvaluationId": "acme_baseline-a1b2c3d4e5",
    "status": "COMPLETED",
    "evaluationResults": {
        "numberOfSessionsCompleted": 47,
        "numberOfSessionsFailed": 3,
        "numberOfSessionsIgnored": 0,
        "totalNumberOfSessions": 50,
        "evaluatorSummaries": [
            {
                "evaluatorId": "Builtin.GoalSuccessRate",
                "statistics": {
                    "averageScore": 0.72
                },
                "totalEvaluated": 47,
                "totalFailed": 0
            },
            {
                "evaluatorId": "Builtin.Helpfulness",
                "statistics": {
                    "averageScore": 0.81
                },
                "totalEvaluated": 47,
                "totalFailed": 0
            }
        ]
    }
}

Per-session detail in CloudWatch Logs

The outputConfig field in the GetBatchEvaluation response specifies a CloudWatch Logs location where per-session, per-evaluator results are written as OpenTelemetry events.

{
    "outputConfig": {
        "cloudWatchConfig": {
            "logGroupName": "/aws/bedrock-agentcore/evaluations/batch",
            "logStreamName": "acme_baseline-a1b2c3d4e5"
        }
    }
}

Each event in the log stream contains per-turn, per-evaluator detail:

To read these events with the Amazon SDK, use the CloudWatch Logs API. agentcore eval batch-evaluation get reads them for you, as described in Viewing results from the CLI.

import boto3

logs_client = boto3.client("logs", region_name="us-west-2")

response = logs_client.get_log_events(
    logGroupName="/aws/bedrock-agentcore/evaluations/batch",
    logStreamName="acme_baseline-a1b2c3d4e5",
)

for event in response["events"]:
    print(event["message"])

Interpreting scores

Batch evaluation scores follow the same conventions as on-demand evaluation:

Error handling

Job-level errors

If the batch evaluation job fails entirely, the status is FAILED and errorDetails contains one or more error messages describing what went wrong. Common causes:

Session-level errors

Individual sessions can fail while the overall job succeeds. The numberOfSessionsFailed count in evaluationResults indicates how many sessions had errors. Per-session errors are recorded in the CloudWatch Logs output.

Evaluator-level errors

Within a successfully evaluated session, individual evaluators can fail. The totalFailed count on each evaluator summary indicates how many sessions that evaluator could not score. Common causes include malformed spans or missing required attributes.

Comparing results across runs

A common workflow is to run batch evaluation before and after a change (prompt update, model swap, tool modification) and compare the aggregate scores:

# After running two batch evaluations
baseline = client.get_batch_evaluation(batchEvaluationId=baseline_id)
treatment = client.get_batch_evaluation(batchEvaluationId=treatment_id)

def average_scores(job):
    # Key on the dimension too: a multi-dimension evaluator has one summary per dimension.
    return {
        (s["evaluatorId"], s.get("evaluationDimensionName")): s["statistics"]["averageScore"]
        for s in job["evaluationResults"]["evaluatorSummaries"]
        if "averageScore" in s.get("statistics", {})
    }

baseline_summaries = average_scores(baseline)
treatment_summaries = average_scores(treatment)

print(f"{'Evaluator':<45} {'Baseline':>10} {'Treatment':>10} {'Delta':>10}")
print("=" * 77)
for (eid, dim), b in baseline_summaries.items():
    t = treatment_summaries.get((eid, dim), 0)
    name = f"{eid}/{dim}" if dim else eid
    print(f"{name:<45} {b:>10.4f} {t:>10.4f} {t - b:>+10.4f}")

Viewing results from the CLI

The AgentCore CLI provides the following commands:

# View a single batch evaluation job and its results
agentcore eval batch-evaluation get \
  --id acme_baseline-a1b2c3d4e5 \
  --json

# List batch evaluation jobs
agentcore eval batch-evaluation list --json