Training documentation

RNN models

class supernnova.training.vanilla_rnn.VanillaRNN(input_size, settings)

Bases: Module

forward(x, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.variational_rnn.VariationalRNN(input_size, settings)

Bases: Module

forward(x, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.variational_rnn.VariationalRecurrentDropout

Bases: Module

This is a renamed Locked Dropout from https://github.com/salesforce/awd-lstm-lm

• We added a mean_field_inference flag to carry out mean field inference

We do this so that choosing whether to use dropout or not at inference time is explicit

forward(x, dropout, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.variational_rnn.VariationalDropout(*args, **kwargs)

Bases: Module

Re-implementation of torch.nn.modules.Dropout

• training flag is always set to True

• We added a mean_field_inference flag to carry out mean field inference

We do this rather than using the training flag so that we can more explicitly decide to use dropout or not at inference time

forward(x, dropout, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.variational_rnn.WeightDrop(module, weights, dropout)

Bases: Module

WeightDrop from https://github.com/salesforce/awd-lstm-lm

• Removed the variational input parameter as we will always use WeightDrop in variational mode

forward(*args, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

supernnova.training.variational_rnn.embedded_dropout(embed, words, dropout, mean_field_inference=False)

embedded_dropout from https://github.com/salesforce/awd-lstm-lm with some modifications like the mean field inference flag

class supernnova.training.bayesian_rnn.BayesianRNN(input_size, settings)

Bases: Module

forward(x, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.bayesian_rnn.BayesRNNBase(mode: str, input_size: int, hidden_size: int, prior: object, num_layers: int = 1, bias: bool = True, batch_first: bool = False, dropout: float = 0, bidirectional: bool = False, mu_lower=-0.05, mu_upper=0.05, rho_lower=-1, rho_upper=-4)

Bases: Module

flatten_parameters() → None

Resets parameter data pointer so that they can use faster code paths.

extra_repr() → str

Set the extra representation of the module

To print customized extra information, you should re-implement this method in your own modules. Both single-line and multi-line strings are acceptable.

class supernnova.training.bayesian_rnn.BayesLSTM(*args, **kwargs)

Bases: BayesRNNBase

forward(input, hx=None, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.bayesian_rnn.BayesGRU(*args, **kwargs)

Bases: BayesRNNBase

class supernnova.training.bayesian_rnn.BayesLinear(in_features, out_features, prior, mu_lower, mu_upper, rho_lower, rho_upper)

Bases: Module

forward(input, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.bayesian_rnn.BayesBiasLinear(in_features, out_features, prior, mu_lower, mu_upper, rho_lower, rho_upper)

Bases: Module

forward(X, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class supernnova.training.bayesian_rnn.BayesEmbedding(num_embeddings, embedding_dim, prior, mu_lower, mu_upper, rho_lower, rho_upper)

Bases: Module

forward(input, mean_field_inference=False)

Defines the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

supernnova.training.bayesian_rnn.logsumexp(x, dim)

Logsumexp trick to avoid overflow in a log of sum of exponential expression

Parameters:

x (Tensor) – the input on which to compute the log of sum of exponential

Returns:

the computed log of sum of exponential

Return type:

logsum (Tensor)

RNN trainer

supernnova.training.train_rnn.get_lr(settings)

Select optimal starting learning rate when training with a 1-cycle policy

Parameters:

settings (ExperimentSettings) – controls experiment hyperparameters

supernnova.training.train_rnn.train_cyclic(settings)

Train RNN models with a 1-cycle policy

Parameters:

settings (ExperimentSettings) – controls experiment hyperparameters

supernnova.training.train_rnn.save_normalizations(settings)

Save normalization used for training

Saves a json file with the normalization used for each feature

Parameters:

settings (ExperimentSettings) – controls experiment hyperparameters

supernnova.training.train_rnn.train(settings)

Train RNN models with a decay on plateau policy

Parameters:

settings (ExperimentSettings) – controls experiment hyperparameters

RandomForest trainer

supernnova.training.train_randomforest.train(settings)

Train random forest models

• Features are stored in a .FITRES file found in settings.processed_dir.

• Carry out a train/val split based on predefined splits

• Compute evaluation metrics on train and validation set

• Save trained models to settings.models_dir

Parameters:

settings – (ExperimentSettings) custom class to hold hyperparameters

Training utilities

supernnova.utils.training_utils.normalize_arr(arr, settings)

Normalize array before input to RNN

• Log transform

• Mean and std dev normalization

Parameters:

• arr (np.array) –

• settings (ExperimentSettings) – controls experiment hyperparameters

Returns:

(np.array) the normalized array

supernnova.utils.training_utils.unnormalize_arr(arr, settings)

UnNormalize array

Parameters:

• arr (np.array) –

• settings (ExperimentSettings) – controls experiment hyperparameters

Returns:

(np.array) the normalized array

supernnova.utils.training_utils.fill_data_list(idxs, arr_data, arr_target, arr_SNID, settings, n_features, desc, test=False)

Utility to create a list of data tuples used as inputs to RNN model

The settings object specifies which feature are selected

Parameters:

• idxs (np.array or list) – idx of data point to select

• arr_data (np.array) – features

• arr_target (np.array) – target

• arr_SNID (np.array) – lightcurve unique ID

• settings (ExperimentSettings) – controls experiment hyperparameters

• n_features (int) – total number of features in arr_data

• desc (str) – message to display while loading

• test (bool) – If True: add more data to the list, as it is required at test time. Default: False

Returns:

(list) the list of data tuples

supernnova.utils.training_utils.load_HDF5(settings, test=False)

Load data from HDF5

Parameters:

• settings (ExperimentSettings) – controls experiment hyperparameters

• test (bool) – If True: load data for test. Default: False

Returns:

list_data_test (list) test data tuples if test is True

or

Tuple containing

• list_data_train (list): training data tuples

• list_data_val (list): validation data tuples

supernnova.utils.training_utils.get_model(settings, input_size)

Create RNN model

Parameters:

• settings (ExperimentSettings) – controls experiment hyperparameters

• input_size (int) – dimension of the input data

Returns:

(torch.nn Model) pytorch model

supernnova.utils.training_utils.get_optimizer(settings, model)

Create gradient descent optimizer

Parameters:

• settings (ExperimentSettings) – controls experiment hyperparameters

• model (torch.nn Model) – the pytorch model

Returns:

(torch.optim) the gradient descent optimizer

supernnova.utils.training_utils.get_data_batch(list_data, idxs, settings, max_lengths=None, OOD=None)

Create a batch in a deterministic way

Parameters:

• list_data – (list) tuples of (X, target, lightcurve_ID)

• idxs – (array / list) indices of batch element in list_data

• settings (ExperimentSettings) – controls experiment hyperparameters

• max_length (int) – Maximum light curve length to be used Default: None.

• OOD (str) – Whether to modify data to create out of distribution data to be used Default: None.

Returns:

Tuple containing

• packed_tensor (torch PackedSequence): the packed features

• X_tensor (torch Tensor): the features

• target_tensor (torch Tensor): the target

supernnova.utils.training_utils.train_step(settings, rnn, packed_tensor, target_tensor, criterion, optimizer, batch_size, num_batches)

Full training step : Forward and Backward pass

Parameters:

• settings (ExperimentSettings) – controls experiment hyperparameters

• rnn (torch.nn Model) – pytorch model to train

• packed_tensor (torch PackedSequence) – input tensor in packed form

• target_tensor (torch Tensor) – target tensor

• criterion (torch loss function) – loss function to optimize

• optimizer (torch optim) – the gradient descent optimizer

• batch_size (int) – batch size

• num_batches (int) – number of minibatches to scale KL cost in Bayesian

supernnova.utils.training_utils.eval_step(rnn, packed_tensor, batch_size)

Eval step: Forward pass only

Parameters:

• rnn (torch.nn Model) – pytorch model to train

• packed_tensor (torch PackedSequence) – input tensor in packed form

• batch_size (int) – batch size

Returns:

output of rnn

Return type:

output (torch Tensor)

supernnova.utils.training_utils.plot_loss(d_train, d_val, epoch, settings)

Plot loss curves

Plot training and validation logloss

Parameters:

• d_train (dict of arrays) – training log losses

• d_val (dict of arrays) – validation log losses

• epoch (int) – current epoch

• settings (ExperimentSettings) – custom class to hold hyperparameters

supernnova.utils.training_utils.get_evaluation_metrics(settings, list_data, model, sample_size=None)

Compute evaluation metrics on a list of data points

Parameters:

• settings (ExperimentSettings) – custom class to hold hyperparameters

• list_data (list) – contains data to evaluate

• model (torch.nn Model) – pytorch model

• sample_size (int) – subset of the data to use for validation. Default: None

Returns:

d_losses (dict) maps metrics to their computed value

supernnova.utils.training_utils.get_loss_string(d_losses_train, d_losses_val)

Obtain a loss string to display training progress

Parameters:

• d_losses_train (dict) – maps {metric:value} for the training data

• d_losses_val (dict) – maps {metric:value} for the validation data

Returns:

the loss string to display

Return type:

loss_str (str)

supernnova.utils.training_utils.save_training_results(settings, d_monitor, training_time)

Obtain a loss string to display training progress

Parameters:

• settings (ExperimentSettings) – controls experiment hyperparameters

• d_monitor (dict) – maps {metric:value}

• training_time (float) – amount of time training took

Returns:

the loss string to display

Return type:

loss_str (str)

supernnova.utils.training_utils.save_randomforest_model(settings, clf)

Save RandomForest model

Parameters:

• settings (ExperimentSettings) – controls experiment hyperparameters

• clf (RandomForestClassifier) – RandomForest model

supernnova.utils.training_utils.load_randomforest_model(settings, model_file=None)

Load RandomForest model

Parameters:

• settings (ExperimentSettings) – controls experiment hyperparameters

• model_file (str) – path to saved randomforest model. Default: None

Returns:

(RandomForestClassifier) RandomForest model

supernnova.utils.training_utils.train_and_evaluate_randomforest_model(clf, X_train, y_train, X_val, y_val)

Train a RandomForestClassifier and evaluate AUC, precision, accuracy on a validation set

Parameters:

• clf (RandomForestClassifier) – RandomForest model to fit and evaluate

• X_train (np.array) – the training features

• y_train (np.array) – the training target

• X_val (np.array) – the validation features

• y_val (np.array) – the validation target

class supernnova.utils.training_utils.StopOnPlateau(patience=10, reduce_lr_on_plateau=False, max_learning_rate_reduction=3)

Bases: object

Detect plateau on accuracy (or any metric) If chosen, will reduce learning rate of optimizer once in the Plateau

Parameters:

• patience (int) – number of epochs to wait, after which we decrease the LR if the validation loss is plateauing

• reduce_lr-on_plateau (bool) – If True, reduce LR after loss has not improved in the last patience epochs

• max_learning_rate_reduction (float) – max factor by which to reduce the learning rate