本文属于机器翻译版本。若本译文内容与英语原文存在差异，则一律以英文原文为准。 # 批量评估入门 本演练将带您从已部署的代理到使用 Acme Store 客户支持代理的批量评估结果。您将创建代理、部署代理、生成示例会话、运行批量评估并读取结果。 **Topics** + [开始前的准备工作](#batch-gs-before-you-begin) + [步骤 1：创建和部署示例代理](#batch-gs-step1) + [步骤 2：生成示例会话](#batch-gs-step2) + [第 3 步：运行批量评估](#batch-gs-step3) + [第 4 步：阅读每个会话的详细信息](#batch-gs-step4) + [后续步骤](#batch-gs-next-steps) ## 开始前的准备工作 请确保： + 已安装 AgentCore CLI (`agentcore --version`) + Amazon 具有和权限`bedrock-agentcore`的证书 `logs` + 中启用了交易搜索 CloudWatch + Python 3.10\+（用于 boto3 示例） 有关完整详细信息，请参阅[先决条件](batch-evaluations-prereqs.md)。 boto3 示例中使用了以下常量。部署代理后，将它们替换为你自己的值： ``` REGION = "us-west-2" AGENT_ARN = "arn:aws:bedrock-agentcore:us-west-2:123456789012:runtime/AcmeSupport-abc123" SERVICE_NAME = "AcmeSupport-abc123.DEFAULT" LOG_GROUP = "/aws/bedrock-agentcore/runtimes/AcmeSupport-abc123-DEFAULT" ``` ## 步骤 1：创建和部署示例代理 创建 AgentCore 项目并将默认代理代码替换为 Acme Store 客户支持代理。该代理有五种工具用于处理订单、退货、运输、折扣和升级。 ### 创建项目 ``` agentcore create --name AcmeSupport --framework Strands --model-provider Bedrock --memory none cd AcmeSupport ``` ### 替换代理代码 打开其内容`app/AcmeSupport/main.py`并将其替换为以下内容： ``` """Acme Store customer support agent.""" from strands import Agent, tool from strands.models.bedrock import BedrockModel from bedrock_agentcore.runtime import BedrockAgentCoreApp app = BedrockAgentCoreApp() MODEL_ID = "global.anthropic.claude-sonnet-4-6" SYSTEM_PROMPT = ( "You are a helpful customer support assistant for Acme Store. " "Help customers with their orders, returns, and shipping questions." ) @tool def lookup_order(order_id: str) -> str: """Look up an order by ID and return its status, item, and delivery details.""" orders = { "ORD-1001": { "status": "delivered", "item": "Blue T-Shirt (L)", "delivered": "2026-03-28", "total": "$29.99", }, "ORD-1002": { "status": "in_transit", "item": "Running Shoes (10)", "shipped": "2026-03-30", "est_delivery": "2026-04-05", "total": "$89.99", }, "ORD-1003": { "status": "delayed", "item": "Wireless Headphones", "shipped": "2026-03-25", "est_delivery": "2026-03-29", "days_late": 5, "total": "$59.99", }, "ORD-1004": { "status": "processing", "item": "Yoga Mat", "ordered": "2026-04-02", "total": "$34.99", }, "ORD-1005": { "status": "delivered", "item": "Coffee Maker", "delivered": "2026-03-20", "total": "$149.99", }, } return str(orders.get(order_id, {"error": f"Order {order_id} not found"})) @tool def initiate_return(order_id: str, reason: str) -> str: """Initiate a return for an order. Sends a return label to the customer.""" return ( f"Return initiated for {order_id}. Reason: {reason}. " "Return label sent to customer email. Please ship within 14 days." ) @tool def check_shipping_status(order_id: str) -> str: """Check detailed shipping status including carrier location and delays.""" statuses = { "ORD-1002": ( "Package is with carrier, currently in Portland OR. " "On schedule for April 5." ), "ORD-1003": ( "Package delayed at distribution center in Memphis TN. " "Original delivery was March 29. Now 5 days late. " "Acme Store policy: orders delayed 3+ days qualify for 15% discount." ), } return statuses.get(order_id, f"No active shipment found for {order_id}.") @tool def apply_discount(order_id: str, discount_percent: int, reason: str) -> str: """Apply a percentage discount to an order and issue a refund.""" return ( f"Applied {discount_percent}% discount to {order_id}. " f"Reason: {reason}. Refund will appear in 3-5 business days." ) @tool def escalate_to_human(reason: str) -> str: """Escalate the conversation to a human support agent.""" return ( f"Escalated to human agent. Reason: {reason}. " "Estimated wait time: 3 minutes." ) agent = Agent( model=BedrockModel(model_id=MODEL_ID), tools=[lookup_order, initiate_return, check_shipping_status, apply_discount, escalate_to_human], system_prompt=SYSTEM_PROMPT, ) @app.entrypoint def invoke(payload, context): result = agent(str(payload.get("prompt", "Hello"))) return {"response": str(result)} if __name__ == "__main__": app.run() ``` ### 部署并验证 ``` agentcore deploy ``` 部署后，验证代理是否正在运行： ``` agentcore invoke --prompt "What's the status of order ORD-1001?" ``` 您应该会看到包含订单详情的回复。请注意来自`agentcore status --json`的运行时 ARN、服务名称和日志组，在 boto3 示例中将需要这些信息。 **注意** 如果您已经在 AgentCore Runtime 上部署了启用了可观察性的代理，请跳过此步骤，在接下来的演练中使用自己的代理。 ## 步骤 2：生成示例会话 使用不同的提示调用代理以创建用于评估的会话。这些提示涵盖了不同的场景：订单查询、退货、发货延迟、折扣请求和多工具互动。 **Example** ``` agentcore invoke --runtime AcmeSupport --prompt "What's the status of my order ORD-1001?" agentcore invoke --runtime AcmeSupport --prompt "I need to return order ORD-1001, the shirt doesn't fit." agentcore invoke --runtime AcmeSupport --prompt "What's the shipping status on ORD-1002?" agentcore invoke --runtime AcmeSupport --prompt "My order ORD-1003 is delayed, can you help?" agentcore invoke --runtime AcmeSupport --prompt "I'd like to check on order ORD-1004 please." agentcore invoke --runtime AcmeSupport --prompt "Can you look up order ORD-1005 for me?" agentcore invoke --runtime AcmeSupport --prompt "I want to return the coffee maker from order ORD-1005, it's defective." agentcore invoke --runtime AcmeSupport --prompt "Where is my order ORD-1002? It should have arrived by now." agentcore invoke --runtime AcmeSupport --prompt "ORD-1003 is really late, I want a discount." agentcore invoke --runtime AcmeSupport --prompt "Can you check order ORD-1001 and tell me when it was delivered?" ``` ``` import boto3 import json import uuid client = boto3.client("bedrock-agentcore", region_name=REGION) prompts = [ "What's the status of my order ORD-1001?", "I need to return order ORD-1001, the shirt doesn't fit.", "What's the shipping status on ORD-1002?", "My order ORD-1003 is delayed, can you help?", "I'd like to check on order ORD-1004 please.", "Can you look up order ORD-1005 for me?", "I want to return the coffee maker from order ORD-1005, it's defective.", "Where is my order ORD-1002? It should have arrived by now.", "ORD-1003 is really late, I want a discount.", "Can you check order ORD-1001 and tell me when it was delivered?", ] for i, prompt in enumerate(prompts): session_id = f"acme-eval-{uuid.uuid4().hex[:12]}" print(f"[{i+1}/10] {prompt[:60]}...") response = client.invoke_agent_runtime( agentRuntimeArn=AGENT_ARN, runtimeSessionId=session_id, payload=json.dumps({"prompt": prompt}).encode(), ) response_body = response["response"].read() print(f" Done (session: {session_id})") print("\nAll sessions created.") ``` 在最后一次调用后等待 2-3 分钟 CloudWatch 以提取遥测数据，然后再继续。 ## 第 3 步：运行批量评估 开始批量评估以对最近的所有会话进行评分。该服务从 CloudWatch 日志中发现会话，针对每个会话运行每个评估器，并返回汇总结果。 **Example** ``` agentcore run batch-evaluation \ --runtime AcmeSupport \ --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \ --wait ``` 默认情况下，`agentcore run batch-evaluation`启动任务并立即返回（不阻塞）。传递`--wait`到区块，直到任务达到终端状态。使用`--wait`，CLI 从您的项目配置中解析 CloudWatch 日志组和服务名称，启动作业，阻塞直到其达到终端状态，然后打印每个评估者的平均分数： ``` Batch evaluation completed: acme-eval-a1b2c3d4 Sessions: 10 completed, 0 failed, 10 total Evaluator Avg Score ───────────────────────────────────────────── Builtin.GoalSuccessRate 0.7200 Builtin.Helpfulness 0.8100 Builtin.Faithfulness 0.8500 Results saved to .cli/jobs/batch-eval-results/ ``` 添加`--json`以发出用于编写脚本的计算机可读结果（包括`batchEvaluationId`和每个评估器`averageScore`），并`-n `对运行进行标记，以便您可以比较不同运行的结果。例如： ``` agentcore run batch-evaluation \ --runtime AcmeSupport \ --evaluator Builtin.GoalSuccessRate Builtin.Helpfulness Builtin.Faithfulness \ -n acme_baseline \ --wait ``` ``` import boto3 import uuid import time import json eval_client = boto3.client("bedrock-agentcore", region_name=REGION) # Start the batch evaluation response = eval_client.start_batch_evaluation( batchEvaluationName=f"acme_baseline_{uuid.uuid4().hex[:8]}", evaluators=[ {"evaluatorId": "Builtin.GoalSuccessRate"}, {"evaluatorId": "Builtin.Helpfulness"}, {"evaluatorId": "Builtin.Faithfulness"}, ], dataSourceConfig={ "cloudWatchLogs": { "serviceNames": [SERVICE_NAME], "logGroupNames": [LOG_GROUP], } }, clientToken=str(uuid.uuid4()), ) batch_eval_id = response["batchEvaluationId"] print(f"Started: {batch_eval_id}") # Poll until complete while True: result = eval_client.get_batch_evaluation(batchEvaluationId=batch_eval_id) status = result["status"] print(f"Status: {status}") if status in ("COMPLETED", "COMPLETED_WITH_ERRORS", "FAILED", "STOPPED"): break time.sleep(30) print(json.dumps(result, indent=4, default=str)) ``` ## 第 4 步：阅读每个会话的详细信息 总分可以告诉你总体情况。要查看各个会话的每回合每个评估者的分数，请使用内置的 CLI 查看命令或直接从日志中读取评估事件。 CloudWatch **Example** CLI 提供一流的命令来查看已完成的批量评估作业及其结果。按批量评估作业 ID 查看特定任务，或列出过去的作业： ``` # View a batch evaluation job and its results agentcore view batch-evaluation acme-eval-a1b2c3d4 # List batch evaluation jobs agentcore batch-evaluations history ``` 当没有给出标志时，这些命令以交互方式运行。`--json`例如，添加非交互式、机器可读的输出。`agentcore view batch-evaluation acme-eval-a1b2c3d4 --json` ``` # Get the output location from the batch evaluation result output = result["outputConfig"]["cloudWatchConfig"] log_group = output["logGroupName"] log_stream = output["logStreamName"] # Read the events logs_client = boto3.client("logs", region_name=REGION) response = logs_client.get_log_events( logGroupName=log_group, logStreamName=log_stream, ) for event in response["events"]: event_attrs = json.loads(event["message"]).get("attributes", {}) print(f"Score: {event_attrs.get('gen_ai.evaluation.score.value')}") print(f"Label: {event_attrs.get('gen_ai.evaluation.score.label')}") print(f"Explanation: {event_attrs.get('gen_ai.evaluation.explanation', '')[:200]}") print() ``` ## 后续步骤 + **筛选会话 **-按 ID 或时间范围评估特定会话。请参见[开始批量评估](batch-evaluations-start.md)。 + **针对数据集运行 **-针对预定义场景调用代理并自动评估结果。请参阅[数据集评估](dataset-evaluations.md)。 + **比较运行次数 **-在变更之前和之后运行批量评估并比较分数。请参阅[了解结果和输出](batch-evaluations-results.md)。