Amazon Web Services 文档中描述的 Amazon Web Services 服务或功能可能因区域而异。要查看适用于中国区域的差异，请参阅 中国的 Amazon Web Services 服务入门 (PDF)。

本文属于机器翻译版本。若本译文内容与英语原文存在差异，则一律以英文原文为准。

基于专栏的评委

概述

Rubric Judge 是一款基于 Nova 2.0 Lite 构建的增强型 LLM-as-a-judge评估模型。与仅提供偏好判决（A>B、B>A 或平局）的原始评委模型不同，Rubric Judge动态生成针对每个提示量身定制的自定义评估标准，并在多个维度上分配精细分数。

关键能力

模型生成的示例标准

price_validation:  
  description: "The response includes validation to ensure price is a positive value."  
  type: "scale"  
  weight: 0.3

该模型根据所有生成的标准评估两个响应，然后使用这些标准级别的分数为其最终的偏好决策提供依据。

配方配置

专栏评委食谱

通过在食谱task: rubric_llm_judge中设置以下内容来启用评分量规判断：

run:  
  name: nova-eval-job-name                              # [MODIFIABLE] Unique identifier for your evaluation job  
  model_type: amazon.nova-2-lite-v1:0:256k              # [FIXED] Rubric Judge model type  
  model_name_or_path: "nova-lite-2/prod"                # [FIXED] Path to model checkpoint or identifier  
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job  
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job  
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job  
    
evaluation:  
  task: rubric_llm_judge                                # [FIXED] Evaluation task - enables Rubric Judge  
  strategy: judge                                       # [FIXED] Evaluation strategy  
  metric: all                                           # [FIXED] Metric calculation method  
    
inference:  
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate  
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter  
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter  
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

原版 LLM as a Judge 食谱（供比较）

最初的评委模型使用task: llm_judge：

run:  
  name: eval-job-name                                   # [MODIFIABLE] Unique identifier for your evaluation job  
  model_type: amazon.nova-micro-v1:0:128k               # [FIXED] Model type   
  model_name_or_path: "nova-micro/prod"                 # [FIXED] Path to model checkpoint or identifier  
  replicas: 1                                           # [MODIFIABLE] Number of replicas for SageMaker Training job  
  data_s3_path: ""                                      # [FIXED] Leave empty for SageMaker Training job  
  output_s3_path: ""                                    # [FIXED] Leave empty for SageMaker Training job  
    
evaluation:  
  task: llm_judge                                       # [FIXED] Original judge task  
  strategy: judge                                       # [FIXED] Evaluation strategy  
  metric: all                                           # [FIXED] Metric calculation method  
  
inference:  
  max_new_tokens: 12000                                 # [MODIFIABLE] Maximum tokens to generate  
  top_k: -1                                             # [MODIFIABLE] Top-k sampling parameter  
  top_p: 1.0                                            # [MODIFIABLE] Nucleus sampling parameter  
  temperature: 0                                        # [MODIFIABLE] Sampling temperature (0 = deterministic)

输入数据集格式

输入数据集的格式与原始判断模型相同：

必填字段

示例数据集（JSONL 格式）

{"prompt": "What is the most effective way to combat climate change?", "response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.", "response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."}  
{"prompt": "Explain how a computer's CPU works", "response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.", "response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."}  
{"prompt": "How does photosynthesis work?", "response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.", "response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."}

格式要求

评估输出

输出结构

与最初的评委模型相比，Rubric Judge 得出的评估指标更强：

{  
  "config_general": {  
    "lighteval_sha": "string",  
    "num_fewshot_seeds": "int",  
    "max_samples": "int | null",  
    "job_id": "int",  
    "start_time": "float",  
    "end_time": "float",  
    "total_evaluation_time_secondes": "string",  
    "model_name": "string",  
    "model_sha": "string",  
    "model_dtype": "string | null",  
    "model_size": "string"  
  },  
  "results": {  
    "custom|rubric_llm_judge_judge|0": {  
      "a_scores": "float",  
      "a_scores_stderr": "float",  
      "b_scores": "float",  
      "b_scores_stderr": "float",  
      "ties": "float",  
      "ties_stderr": "float",  
      "inference_error": "float",  
      "inference_error_stderr": "float",  
      "score": "float",  
      "score_stderr": "float",  
      "weighted_score_A": "float",  
      "weighted_score_A_stderr": "float",  
      "weighted_score_B": "float",  
      "weighted_score_B_stderr": "float",  
      "score_margin": "float",  
      "score_margin_stderr": "float",  
      "winrate": "float",  
      "lower_rate": "float",  
      "upper_rate": "float"  
    }  
  },  
  "versions": {  
    "custom|rubric_llm_judge_judge|0": "int"  
  }  
}

Rubric Judge 中的新指标

以下六个指标是 Rubric Judge 所独有的，可提供精细的质量评估：

了解加权分数指标

目的：加权分数提供连续的质量测量，以补充二元偏好判断，从而可以更深入地了解模型性能。

与原评委的主要区别

解释 score_margin

使用案例

计算方法

加权分数通过以下过程计算：

示例：如果 response_A 的加权总和为 0.65，response_B 的加权和为 0.78，则应为 -0.13，这表明 response_B 在score_margin所有加权标准中的质量要高 13 个百分点。