RFT on Nova 2.0

RFT training data follows the OpenAI conversational format. Each training example is a JSON object containing messages, reference answers, and optional tool definitions. This section provides guidance on preparing effective training data for RFT on Nova 2.0.

Data format and structure

Each training example is a JSON object containing the following:

Each example should be on a single line in your JSONL file, with one JSON object per line.

{  
  "id": "chem-001",  
  "messages": [  
    {  
      "role": "system",  
      "content": "You are a helpful chemistry assistant"  
    },  
    {  
      "role": "user",  
      "content": "Predict hydrogen bond donors and acceptors for this SMILES: CCN(CC)CCC(=O)c1sc(N)nc1C"  
    }  
  ],  
  "reference_answer": {  
    "donor_bond_counts": 2,  
    "acceptor_bond_counts": 4,  
    "explanation": "Calculated using Lipinski's rule of five: N-H groups (2 donors), N and O atoms with lone pairs (4 acceptors)"  
  }  
}

{  
  "id": "math-001",  
  "messages": [  
    {  
      "role": "system",  
      "content": "You are a math tutor"  
    },  
    {  
      "role": "user",  
      "content": "Solve: 2x + 5 = 13"  
    }  
  ],  
  "reference_answer": {  
    "solution": "x = 4",  
    "steps": ["2x = 13 - 5", "2x = 8", "x = 4"]  
  }  
}

{  
  "id": "tool-001",  
  "messages": [  
    {  
      "role": "system",  
      "content": "You are a helpful game master assistant"  
    },  
    {  
      "role": "user",  
      "content": "Generate a strength stat for a warrior character. Apply a +2 racial bonus modifier."  
    }  
  ],  
  "tools": [  
    {  
      "type": "function",  
      "function": {  
        "name": "StatRollAPI",  
        "description": "Generates character stats by rolling 4d6, dropping the lowest die result, and applying a modifier.",  
        "parameters": {  
          "type": "object",  
          "properties": {  
            "modifier": {  
              "description": "An integer representing the modifier to apply to the total of the stat roll.",  
              "type": "integer"  
            }  
          },  
          "required": ["modifier"]  
        }  
      }  
    }  
  ],  
  "reference_answer": {  
    "tool_called": "StatRollAPI",  
    "tool_parameters": {  
      "modifier": 2  
    },  
    "expected_behavior": "Call StatRollAPI with modifier=2 and return the calculated stat value"  
  }  
}

Field descriptions

Additional properties

The "additionalProperties": true setting allows you to include custom fields beyond the core schema requirements, providing flexibility to add any data your reward function needs for proper evaluation.

Common additional fields

You can include the following types of additional fields:

Metadata:

Evaluation criteria:

Example with additional properties

The following example includes custom metadata fields:

{  
  "id": "algebra_001",  
  "messages": [  
    {  
      "role": "system",  
      "content": "You are a math tutor"  
    },  
    {  
      "role": "user",  
      "content": "Solve: 2x + 5 = 13"  
    }  
  ],  
  "reference_answer": {  
    "solution": "x = 4",  
    "steps": ["2x = 13 - 5", "2x = 8", "x = 4"]  
  },  
  "task_id": "algebra_001",  
  "difficulty_level": "easy",  
  "domain": "algebra",  
  "expected_reasoning_steps": 3  
}

Dataset size recommendations

Starting point

Begin with the following minimum dataset sizes:

Prioritize high-quality input data and a reliable reward function that executes consistently on model responses.

Evaluation-first approach

Before investing in large-scale RFT training, evaluate your model's baseline performance:

This evaluation-first approach ensures your reward function is bug-free and determines if RFT is the right method for your use case. Starting small allows you to get comfortable with the RFT workflow, identify and fix issues early, validate your approach before scaling up, and test reward function reliability. Once validated, you can expand to larger datasets to further improve performance.

Characteristics of effective training data

Clarity and consistency

Good RFT examples require clear, unambiguous input data that enables accurate reward calculation across different model outputs. Avoid noise in your data, including:

Any ambiguity will mislead the training process and cause the model to learn unintended behaviors.

Diversity

Your dataset should capture the full diversity of production use cases to ensure robust real-world performance. Include:

This diversity helps prevent overfitting and ensures the model handles unfamiliar inputs gracefully.

Reward function considerations

Design your reward function for efficient training:

Fast, scalable reward functions enable rapid iteration and cost-effective experimentation at scale.