评估配方示例

这些配方让您能够通过一套全面的纯文本基准测试来评估 Amazon Nova 模型的基本功能。它们以 xxx_general_text_benchmark_eval.yaml 格式提供。

这些配方让您能够使用自己的数据集进行基准测试，并使用不同类型的指标将模型输出与参考答案进行对比。它们以 xxx_bring_your_own_dataset_eval.yaml 格式提供。

以下是自带数据集的要求：

以下是自带数据集示例条目

{
   "system":"You are a english major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}
{
   "system":"You are a pattern analysis specialist that provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}
{
   "system":"You have great attention to detail that follows instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: gen_qa 
  strategy: gen_qa 
  metric: all

适用以下限制：

Amazon Nova LLM-as-a-Judge 是一项模型评测特征，让客户可以使用自定义数据集，将一个模型的响应质量与基准模型的响应质量进行比较。它接受包含提示、基准响应和挑战响应的数据集，然后使用 Nova Judge 模型基于 Bradley-Terry 概率和成对比较提供胜率指标。

配方以 xxx_llm_judge_eval.yaml 格式提供。

以下是 LLM-as-a-Judge 的要求：

这是一个 LLM-as-a-Judge 示例条目

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

适用以下限制：

适用于多模态的 Nova LLM Judge（图像）是 Nova MM_LLM Judge 的缩写，是一项模型评估功能，可让您使用自定义数据集将来自一个模型的响应质量与基线模型的响应质量进行比较。它接受包含提示、基线响应和挑战者响应的数据集以及 Base64 编码字符串形式的图像，然后使用 Nova Judge 模型通过配对比较提供基于 B radley- Terry 概率的胜率指标。配方格式：xxx_mm_llm_judge _eval.yaml。

Nova LLM 数据集要求

文件格式：

架构格式：.jsonl 文件中的每一行都必须是包含以下字段的 JSON 对象。

示例条目

为了便于阅读，以下示例包括换行符和缩进，但在实际数据集中，每条记录都应在一行上。

{
  "prompt": "what is in the image?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    }
  ],
  "response_A": "a dog.",
  "response_B": "a cat.",
} 
{
  "prompt": "how many animals in echo of the images?",
  "images": [
    {
      "data": "data:image/jpeg;Base64,/9j/2wBDAAQDAwQDAwQEAwQFBAQFBgo..."
    },
    {
      "data": "data:image/jpeg;Base64,/DKEafe3gihn..."
    }
  ],
  "response_A": "The first image contains one cat and the second image contains one dog",
  "response_B": "The first image has one aminal and the second has one animal",
}           

要使用您的自定义数据集，请按以下必填字段修改您的评估配方，且不要更改其中任何内容：

evaluation:
  task: mm_llm_judge
  strategy: judge
  metric: all

限制

Rubric Judge 是一款基于 Nova 2.0 Lite 构建的增强型 LLM-as-a-judge评估模型。与仅提供偏好判决（A>B、B>A 或平局）的原始评委模型不同，Rubric Judge动态生成针对每个提示量身定制的自定义评估标准，并在多个维度上分配精细分数。

关键能力：

模型生成的标准示例：

price_validation:
  description: "The response includes validation to ensure price is a positive value."
  type: "scale"
  weight: 0.3

该模型根据所有生成的标准评估两个响应，然后使用这些标准级别的分数为其最终的偏好决策提供依据。

配方配置

专栏评委食谱

通过在食谱task: rubric_llm_judge中设置以下内容来启用评分量规判断：

run:
  name: nova-eval-job-name                              # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k              # [FIXED] Rubric Judge model type
  model_name_or_path: "nova-lite-2/prod"                # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: rubric_llm_judge                                # [FIXED] Evaluation task - enables Rubric Judge
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

原版 LLM as a Judge 食谱（供比较）

最初的评委模型使用task: llm_judge：

run:
  name: eval-job-name                                   # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-micro-v1:0:128k               # [FIXED] Model type
  model_name_or_path: "nova-micro/prod"                 # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job

evaluation:
  task: llm_judge                                       # [FIXED] Original judge task
  strategy: judge                                       # [FIXED] Evaluation strategy
  metric: all                                           # [FIXED] Metric calculation method

inference:
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

输入数据集格式

输入数据集的格式与原始判断模型相同：

必填字段：

示例数据集（JSONL 格式）：

{"prompt": "What is the most effective way to combat climate change?", "response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.", "response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."}
{"prompt": "Explain how a computer's CPU works", "response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.", "response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."}
{"prompt": "How does photosynthesis work?", "response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.", "response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."}

格式要求：

评估输出

输出结构

与最初的评委模型相比，Rubric Judge 得出的评估指标更强：

{
  "config_general": {
    "lighteval_sha": "string",
    "num_fewshot_seeds": "int",
    "max_samples": "int | null",
    "job_id": "int",
    "start_time": "float",
    "end_time": "float",
    "total_evaluation_time_secondes": "string",
    "model_name": "string",
    "model_sha": "string",
    "model_dtype": "string | null",
    "model_size": "string"
  },
  "results": {
    "custom|rubric_llm_judge_judge|0": {
      "a_scores": "float",
      "a_scores_stderr": "float",
      "b_scores": "float",
      "b_scores_stderr": "float",
      "ties": "float",
      "ties_stderr": "float",
      "inference_error": "float",
      "inference_error_stderr": "float",
      "score": "float",
      "score_stderr": "float",
      "weighted_score_A": "float",
      "weighted_score_A_stderr": "float",
      "weighted_score_B": "float",
      "weighted_score_B_stderr": "float",
      "score_margin": "float",
      "score_margin_stderr": "float",
      "winrate": "float",
      "lower_rate": "float",
      "upper_rate": "float"
    }
  },
  "versions": {
    "custom|rubric_llm_judge_judge|0": "int"
  }
}

Rubric Judge 中的新指标

以下六个指标是 Rubric Judge 所独有的，可提供精细的质量评估：

了解加权分数指标

目的：加权分数提供连续的质量测量，以补充二元偏好判断，从而可以更深入地了解模型性能。

与@@ 原评委的主要区别：

解释 score_margin：

用例：

计算方法

加权分数通过以下过程计算：

示例：如果 response_A 的加权总和为 0.65，response_B 的加权和为 0.78，则应为 -0.13，这表明 response_B 在score_margin所有加权标准中的质量要高 13 个百分点。

推理模型支持

推理模型支持允许使用具有推理功能的 Nova 模型进行评估，这些模型在生成最终响应之前执行明确的内部推理。此功能通过reasoning_effort参数使用 API 级控制来动态启用或禁用推理功能，从而有可能提高复杂分析任务的响应质量。

支持的型号：

配方配置

通过将reasoning_effort参数添加到食谱的inference部分来启用推理：

run:
  name: eval-job-name                                    # [MODIFIABLE] Unique identifier for your evaluation job
  model_type: amazon.nova-2-lite-v1:0:256k               # [FIXED] Must be a reasoning-supported model
  model_name_or_path: nova-lite-2/prod                   # [FIXED] Path to model checkpoint or identifier
  replicas: 1                                            # [MODIFIABLE] Number of replicas for SageMaker Training job
  data_s3_path: ""                                       # [MODIFIABLE] Leave empty for SageMaker Training job; optional for SageMaker HyperPod job
  output_s3_path: ""                                     # [MODIFIABLE] Output path for SageMaker HyperPod job (not compatible with SageMaker Training jobs)

evaluation:
  task: mmlu                                             # [MODIFIABLE] Evaluation task
  strategy: generate                                     # [MODIFIABLE] Evaluation strategy
  metric: all                                            # [MODIFIABLE] Metric calculation method

inference:
  reasoning_effort: high                                 # [MODIFIABLE] Enables reasoning mode; options: low/medium/high or null to disable
  max_new_tokens: 200                                    # [MODIFIABLE] Maximum tokens to generate
  top_k: 50                                              # [MODIFIABLE] Top-k sampling parameter
  top_p: 1.0                                             # [MODIFIABLE] Nucleus sampling parameter
  temperature: 0                                         # [MODIFIABLE] Sampling temperature (0 = deterministic)

使用 reasing_effort 参数

该reasoning_effort参数控制具有推理能力的模型的推理行为。

先决条件：

可用选项：

何时启用推理

使用推理模式（lowmedium、或high）执行以下操作：

使用非推理模式（省略参数）：

问题排查

错误：“推理模式已启用，但模型不支持推理”

原因：reasoning_effort参数设置为非空值，但指定的值model_type不支持推理。

解决方法：