Customizing model hyperparameter configurations in Neptune ML

Tier-1 hyperparameters have the highest precedence. If you set maxHPONumberOfTrainingJobs to a value less than 10, only Tier-1 hyperparameters are tuned, and the rest take their default values.

Tier-2 hyperparameters have lower precedence, so if you have more than 10 but less than 50 total training jobs for a tuning job, then both Tier-1 and Tier-2 hyperparameters are tuned.

Tier 3 hyperparameters are tuned together with Tier-1 and Tier-2 only if you have more than 50 total training jobs.

Finally, fixed hyperparameters are not tuned at all, and always take their default values.

model   –   (String) The model type to be trained (do not modify). Valid values are:

As a rule, don't directly modify the model value, because different model types often have substantially different applicable hyperparameters, which can result in a parsing error after the training job has started.

To change the model type, use the modelName parameter in the modelTraining API rather than change it in the model-HPO-configuration.json file.

A way to change the model type and make fine-grain hyperparameter changes is to copy the default model configuration template for the model that you want to use and paste that into the model-HPO-configuration.json file. There is a folder named hpo-configuration-templates in the same Amazon S3 location as the model-HPO-configuration.json file if the inferred task type supports multiple models. This folder contains all the default hyperparameter configurations for the other models that are applicable to the task.

For example, if you want to change the model and hyperparameter configurations for a KGE link-prediction task from the default transe model to a distmult model, simply paste the contents of the hpo-configuration-templates/distmult.json file into the model-HPO-configuration.json file and then edit the hyperparameters as necessary.

task_type   –   (String) The machine learning task type inferred by or passed directly to the data-processing job (do not modify). Valid values are:

The data-processing job infers the task type by examining the exported dataset and the generated training-job configuration file for properties of the dataset.

This value should not be changed. If you want to train a different task, you need to run a new data-processing job. If the task_type value is not what you were expecting, you should check the inputs to your data-processing job to make sure that they are correct. This includes parameters to the modelTraining API, as well as in the training-job configuration file generated by the data-export process.

eval_metric   –   (String) The evaluation metric should be used for evaluating the model performance and for selecting the best-performing model across HPO runs. Valid values are:

For example, to change the metric to F1, change the eval_metric value as follows:

"  eval_metric": {
    "metric": "F1",
  },

Or, to change the metric to a topk accuracy score, you would change eval_metric as follows:

  "eval_metric": {
    "metric": "acc_topk",
    "topk": 2
  },

eval_frequency   –   (Object) Specifies how often during training the performance of the model on the validation set should be checked. Based on the validation performance, early stopping can then be initiated and the best model can be saved.

The eval_frequency object contains two elements, namely "type" and "value". For example:

  "eval_frequency": {
    "type":  "evaluate_every_pct",
    "value":  0.1
  },

Valid type values are:

The default setting for eval_frequency is:

  "eval_frequency": {
    "type":  "evaluate_every_epoch",
    "value":  1
  },

1-tier-param   –   (Required) An array of Tier-1 hyperparameters.

If you don't want to tune any hyperparameters, you can set this to an empty array. This does not affect the total number of training jobs launched by the SageMaker hyperparameter tuning job. It just means that all training jobs, if there is more than 1 but less than 10, will run with the same set of hyperparameters.

On the other hand, if you want to treat all your tunable hyperparameters with equal significance then you can put all the hyperparameters in this array.

2-tier-param   –   (Required) An array of Tier-2 hyperparameters.

These parameters are only tuned if maxHPONumberOfTrainingJobs has a value greater than 10. Otherwise, they are fixed to the default values.

If you have a training budget of at most 10 training jobs or don't want Tier-2 hyperparameters for any other reason, but you want to tune all tunable hyperparameters, you can set this to an empty array.

3-tier-param   –   (Required) An array of Tier-3 hyperparameters.

These parameters are only tuned if maxHPONumberOfTrainingJobs has a value greater than 50. Otherwise, they are fixed to the default values.

If you don't want Tier-3 hyperparameters, you can set this to an empty array.

fixed-param   –   (Required) An array of fixed hyperparameters that take only their default values and do not vary in different training jobs.

If you want to vary all hyperparameters, you can set this to an empty array and either set the value for maxHPONumberOfTrainingJobs large enough to vary all tiers or make all hyperparameters Tier-1.

param   –   (String) The name of the hyperparameter (do not change).

See the list of valid hyperparameter names in Neptune ML.

type   –   (String) The hyperparameter type (do not change).

Valid types are: bool, int, and float.

default   –   (String) The default value for the hyperparameter.

You can set a new default value.

range   –   (Array) The range for a continuous tunable hyperparameter.

This should be an array with two values, namely the minimum and maximum of the range ([min, max]).

options   –   (Array) The options for a categorical tunable hyperparameter.

This array should contain all the options to consider:

  "options" : [value1, value2, ... valuen]

inc_strategy   –   (String) The type of incremental change for continuous tunable hyperparameter ranges (do not change).

Valid values are log, linear, and power2. This applies only when the range key is set.

Modifying this may result in not using the full range of your hyperparameter for tuning.

inc_val   –   (Float) The amount by which successive increments differ for continuous tunablehyperparameters (do not change).

This applies only when the range key is set.

Modifying this may result in not using the full range of your hyperparameter for tuning.

node_strategy   –   (String) Indicates that the effective range for this hyperparameter should change based on the number of nodes in the graph (do not change).

Valid values are "perM" (per million), "per10M" (per 10 million), and "per100M" (per 100 million).

Rather than change this value, change the range instead.

edge_strategy   –   (String) Indicates that the effective range for this hyperparameter should change based on the number of edges in the graph (do not change).

Valid values are "perM" (per million), "per10M" (per 10 million), and "per100M" (per 100 million).

Rather than change this value, change the range instead.

batch-size   –   The size of the batch of target nodes using in one forward pass. Type: int.

Setting this to a much larger value can cause memory issues for training on GPU instances.

concat-node-embed   –   Indicates whether to get the initial representation of a node by concatenating its processed features with learnable initial node embeddings in order to increase the expressivity of the model. Type: bool.

dropout   –   The dropout probability applied to dropout layers. Type: float.

edge-num-hidden   –   The hidden layer size or number of units for the edge feature module. Only used when use-edge-features is set to True. Type: float.

enable-early-stop   –   Toggles whether or not to use the early stopping feature. Type: bool. Default: true.

Use this Boolean parameter to turn off the early stop feature.

fanout   –   The number of neighbors to sample for a target node during neighbor sampling. Type: int.

This value is tightly coupled with num-layers and should always be in the same hyperparameter tier. This is because you can specify a fanout for each potential GNN layer.

Because this hyperparameter can cause model performance to vary widely, it should be fixed or set as a Tier-2 or Tier-3 hyperparameter. Setting it to a large value can cause memory issues for training on GPU instance.

gamma   –   The margin value in the score function. Type: float.

This applies to KGE link-prediction models only.

l2norm   –   The weight decay value used in the optimizer which imposes an L2 normalization penalty on the weights. Type: bool.

layer-norm   –   Indicates whether to use layer normalization for rgcn models. Type: bool.

low-mem   –   Indicates whether to use a low-memory implementation of the relation message passing function at the expense of speed. Type: bool.

lr   –   The learning rate. Type: float.

This should be set as a Tier-1 hyperparameter.

neg-share   –   In link prediction, indicates whether positive sampled edges can share negative edge samples. Type: bool.

num-bases   –   The number of bases for basis decomposition in a rgcn model. Using a value of num-bases that is less than the number of edge types in the graph acts as a regularizer for the rgcn model. Type: int.

num-epochs   –   The number of epochs of training to run. Type: int.

An epoch is a complete training pass through the graph.

num-hidden   –   The hidden layer size or number of units. Type: int.

This also sets the initial embedding size for featureless nodes.

Setting this to a much larger value without reducing batch-size can cause out-of-memory issues for training on GPU instance.

num-layer   –   The number of GNN layers in the model. Type: int.

This value is tightly coupled with the fanout parameter and should come after fanout is set in the same hyperparameter tier.

Because this can cause model performance to vary widely, it should be fixed or set as a Tier-2 or Tier-3 hyperparameter.

num-negs   –   In link prediction, the number of negative samples per positive sample. Type: int.

per-feat-name-embed   –   Indicates whether to embed each feature by independently transforming it before combining features. Type: bool.

When set to true, each feature per node is independently transformed to a fixed dimension size before all the transformed features for the node are concatenated and further transformed to the num_hidden dimension.

When set to false, the features are concatenated without any feature-specific transformations.

regularization-coef   –   In link prediction, the coefficient of regularization loss. Type: float.

rel-part   –   Indicates whether to use relation partition for KGE link prediction. Type: bool.

sparse-lr   –   The learning rate for learnable-node embeddings. Type: float.

Learnable initial node embeddings are used for nodes without features or when concat-node-embed is set. The parameters of the sparse learnable node embedding layer are trained using a separate optimizer which can have a separate learning rate.

use-class-weight   –   Indicates whether to apply class weights for imbalanced classification tasks. If set to to true, the label counts are used to set a weight for each class label. Type: bool.

use-edge-features   –   Indicates whether to use edge features during message passing. If set to true, a custom edge feature module is added to the RGCN layer for edge types that have features. Type: bool.

use-self-loop   –   Indicates whether to include self loops in training a rgcn model. Type: bool.

window-for-early-stop   –   Controls the number of latest validation scores to average to decide on an early stop. The default is 3. type=int. See also Early stopping of the model training process in Neptune ML. Type: int. Default: 3.

See .

Edit the minimum and/or maximum values of range hyperparameters.

Set a hyperparameter to a fixed value by moving it to the fixed-param section and setting its default value to the fixed value you want it to take.

Change the priority of a hyperparameter by placing it in a particular tier, editing its range, and making sure that its default value is set appropriately.