Services or capabilities described in Amazon Web Services documentation might vary by Region. To see the differences applicable to the China Regions, see Getting Started with Amazon Web Services in China (PDF).

Evaluating your trained model

An evaluation recipe is a YAML configuration file that defines how your Amazon Nova model evaluation job is executed. With this recipe, you can assess the performance of a base or trained model against common benchmarks or your own custom datasets. Metrics can be stored in Amazon S3 or TensorBoard. The evaluation provides quantitative metrics that help you assess model performance across various tasks to determine if further customization is needed.

Model evaluation is an offline process, where models are tested against fixed benchmarks with predefined answers. They are not assessed in real-time or against live user interactions. For real-time evaluations, you can evaluate the model after it is deployed to Amazon Bedrock by calling the Amazon Bedrock runtime APIs.

Available benchmark tasks

A sample code package is available that demonstrates how to calculate benchmark metrics using the SageMaker AI model evaluation feature for Amazon Nova. To access the code packages, see sample-Nova-lighteval-custom-task.

Here is a list of the supported, available industry standard benchmarks. You can specify the following benchmarks in the eval_task parameter:

The following mmlu subtasks are available:

MMLU_SUBTASKS = [
    "abstract_algebra",
    "anatomy",
    "astronomy",
    "business_ethics",
    "clinical_knowledge",
    "college_biology",
    "college_chemistry",
    "college_computer_science",
    "college_mathematics",
    "college_medicine",
    "college_physics",
    "computer_security",
    "conceptual_physics",
    "econometrics",
    "electrical_engineering",
    "elementary_mathematics",
    "formal_logic",
    "global_facts",
    "high_school_biology",
    "high_school_chemistry",
    "high_school_computer_science",
    "high_school_european_history",
    "high_school_geography",
    "high_school_government_and_politics",
    "high_school_macroeconomics",
    "high_school_mathematics",
    "high_school_microeconomics",
    "high_school_physics",
    "high_school_psychology",
    "high_school_statistics",
    "high_school_us_history",
    "high_school_world_history",
    "human_aging",
    "human_sexuality",
    "international_law",
    "jurisprudence",
    "logical_fallacies",
    "machine_learning",
    "management",
    "marketing",
    "medical_genetics",
    "miscellaneous",
    "moral_disputes",
    "moral_scenarios",
    "nutrition",
    "philosophy",
    "prehistory",
    "professional_accounting",
    "professional_law",
    "professional_medicine",
    "professional_psychology",
    "public_relations",
    "security_studies",
    "sociology",
    "us_foreign_policy",
    "virology",
    "world_religions"
]

The following bbh subtasks are available:

BBH_SUBTASKS = [
    "boolean_expressions",
    "causal_judgement",
    "date_understanding",
    "disambiguation_qa",
    "dyck_languages",
    "formal_fallacies",
    "geometric_shapes",
    "hyperbaton",
    "logical_deduction_five_objects",
    "logical_deduction_seven_objects",
    "logical_deduction_three_objects",
    "movie_recommendation",
    "multistep_arithmetic_two",
    "navigate",
    "object_counting",
    "penguins_in_a_table",
    "reasoning_about_colored_objects",
    "ruin_names",
    "salient_translation_error_detection",
    "snarks",
    "sports_understanding",
    "temporal_sequences",
    "tracking_shuffled_objects_five_objects",
    "tracking_shuffled_objects_seven_objects",
    "tracking_shuffled_objects_three_objects",
    "web_of_lies",
    "word_sorting"
]

The following math subtasks are available:

MATH_SUBTASKS = [
    "algebra",
    "counting_and_probability",
    "geometry",
    "intermediate_algebra",
    "number_theory",
    "prealgebra",
    "precalculus",
]

Understanding the recipe parameters

Run configuration

The following is a general run configuration and an explanation of the parameters involved.

run:
  name: eval_job_name 
  model_type: amazon.nova-micro-v1:0:128k 
  model_name_or_path: nova-micro/prod 
  replicas: 1 
  data_s3_path: ""
  output_s3_path: s3://output_path

Evaluation configuration

The following is a model evaluation configuration and an explanation of the parameters involved.

evaluation:
  task: mmlu 
  strategy: zs_cot 
  subtask: mathematics
  metric: accuracy

Inference configuration

The following is an inference configuration and an explanation of the parameters involved. All parameters are optional.

inference:
  max_new_tokens: 200 
  top_k: -1 
  top_p: 1.0 
  temperature: 0

Note that for humaneval, we recommend the following inference configuration:

inference:
  top_k: 1
  max_new_tokens: 1600
  temperature: 0.0

Evaluation recipe examples

Amazon Nova provides four types of evaluation recipes, which are available in the HyperPod recipes GitHub repository.

General text benchmark recipes

These recipes enable you to evaluate the fundamental capabilities of Amazon Nova models across a comprehensive suite of text-only benchmarks. They are provided in the format xxx_general_text_benchmark_eval.yaml.

General multi-modal benchmark recipe

These recipes enable you to evaluate the fundamental capabilities of Amazon Nova models across a comprehensive suite of multi-modality benchmarks. They are provided in the format xxx_general_multi_modal_benchmark_eval.yaml.

Bring your own dataset benchmark recipe

These recipes enable you to bring your own dataset for benchmarking and compare model outputs to reference answers using different types of metrics. They are provided in the format xxx_bring_your_own_dataset_eval.yaml.

The following are the bring your own dataset requirements:

Here is a bring your own data set example entry

{
   "system":"You are a english major with top marks in class who likes to give minimal word responses: ",
   "query":"What is the symbol that ends the sentence as a question",
   "response":"?"
}
{
   "system":"You are a pattern analysis specialist that provides succinct answers: ",
   "query":"What is the next number in this series? 1, 2, 4, 8, 16, ?",
   "response":"32"
}
{
   "system":"You have great attention to detail that follows instructions accurately: ",
   "query":"Repeat only the last two words of the following: I ate a hamburger today and it was kind of dry",
   "response":"of dry"
}

To use your custom dataset, modify your evaluation recipe with the following required fields, do not change any of the content:

evaluation:
  task: gen_qa 
  strategy: gen_qa 
  metric: all

The following limitations apply:

Amazon Nova LLM as a Judge benchmark recipe

Amazon Nova LLM as a Judge is a model evaluation feature that enables customers to compare the quality of responses from one model to a baseline model response on a custom dataset. It takes in a dataset with prompts, baseline responses, and challenger responses, and uses a Nova Judge model to provide a winrate metric based on Bradley-Terry probability with pairwise comparisons.

The recipes are provided in the format xxx_llm_judge_eval.yaml.

The following are the LLM as a Judge requirements:

Here is an LLM as a judge example entry

{
"prompt": "What is the most effective way to combat climate change?",
"response_A": "The most effective way to combat climate change is through a combination of transitioning to renewable energy sources and implementing strict carbon pricing policies. This creates economic incentives for businesses to reduce emissions while promoting clean energy adoption.",
"response_B": "We should focus on renewable energy. Solar and wind power are good. People should drive electric cars. Companies need to pollute less."
}
{
"prompt": "Explain how a computer's CPU works",
"response_A": "CPU is like brain of computer. It does math and makes computer work fast. Has lots of tiny parts inside.",
"response_B": "A CPU (Central Processing Unit) functions through a fetch-execute cycle, where instructions are retrieved from memory, decoded, and executed through its arithmetic logic unit (ALU). It coordinates with cache memory and registers to process data efficiently using binary operations."
}
{
"prompt": "How does photosynthesis work?",
"response_A": "Plants do photosynthesis to make food. They use sunlight and water. It happens in leaves.",
"response_B": "Photosynthesis is a complex biochemical process where plants convert light energy into chemical energy. They utilize chlorophyll to absorb sunlight, combining CO2 and water to produce glucose and oxygen through a series of chemical reactions in chloroplasts."
}

To use your custom dataset, modify your evaluation recipe with the following required fields, don't change any of the content:

evaluation:
  task: llm_judge
  strategy: judge
  metric: all

The following limitations apply:

Start an evaluation job

The following provides a suggested evaluation instance type and model type configuration:

# Install Dependencies (Helm - https://helm.sh/docs/intro/install/)
curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh
rm -f ./get_helm.sh  

# Install the HyperPod CLI
git clone --recurse-submodules https://github.com/aws/sagemaker-hyperpod-cli.git
git checkout -b release_v2
cd sagemaker-hyperpod-cli
pip install .

# Verify the installation
hyperpod --help

# Connect to a HyperPod Cluster
hyperpod connect-cluster --cluster-name cluster-name


# Submit the Job using the recipe for eval
# Namespace by default should be kubeflow
hyperpod start-job [--namespace namespace] --recipe evaluation/nova/nova_micro_p5_48xl_general_text_benchmark_eval --override-parameters \
'{
    "instance_type":"p5d.48xlarge",
    "container": "708977205387.dkr.ecr.us-east-1.amazonaws.com/nova-evaluation-repo:SM-HP-Eval-latest",
    "recipes.run.name": custom-run-name,
    "recipes.run.model_type": model_type,
    "recipes.run.model_name_or_path" " model name or finetune checkpoint s3uri,
    "recipes.run.data_s3_path": s3 for input data only for genqa and llm_judge, must be full S3 path that include filename,
}'

# List jobs
hyperpod list-jobs [--namespace namespace] [--all-namespaces]

# Getting Job details
hyperpod get-job --job-name job-name [--namespace namespace] [--verbose]

# Listing Pods
hyperpod list-pods --job-name job-name --namespace namespace

# Cancel Job
hyperpod cancel-job --job-name job-name [--namespace namespace]

You should also be able to view the job status through Amazon EKS cluster console.

Access and visualize your results

After your evaluation job completes successfully, you can access and analyze the results using the information in this section. Based on the output_s3_path (such as s3://output_path/) defined in the recipe, the output structure is the following:

job_name/
├── eval-result/
│    └── job-name/
│         └── results_[timestamp].json
│         └── details/
│              └── model/
│                  └── execution-date-time/
│                             └──details_task_name_#_datetime.parquet
└── eval-tensorboard-result/
    └── job-name/
        └── eval/
            └── events.out.tfevents.[timestamp]

Metrics results are stored in the specified S3 output location s3://output_path/job_name/eval-result/result-timestamp.json

Tensorboard results are stored in the S3 path s3://output_path/job_name/eval-tensorboard-result/eval/event.out.tfevents.epoch+ip.

Inference outputs, expect for strong_reject and gen_qa, are stored in the S3 path s3://output_path/job_name/eval-result/details/model/taskname.parquet.

To visualize your evaluation metrics in Tensorboard, complete the following steps:

The time series, scalars, and text visualizations are available.

We recommend the following best practices:

You can find HyperPod job error logs in the CloudWatch log group /aws/sagemaker/Clusters/cluster-id.