Validation documentation

Validation RNN

Metrics

supernnova.validation.metrics.aggregate_metrics(settings)[source]

Aggregate all pre-computed METRICS files into a single dataframe for analysis

Save a csv dataframe aggregating all the metrics

Parameters:

settings (ExperimentSettings) – custom class to hold hyperparameters

supernnova.validation.metrics.get_metrics_singlemodel(settings, prediction_file=None, model_type='rnn')[source]

Launch computation of all evaluation metrics for a given model, specified by the settings object or by a model file

Save a pickled dataframe (we pickle because we’re saving numpy arrays, which are not easily savable with the to_csv method).

Parameters:

  • settings (ExperimentSettings) – custom class to hold hyperparameters

  • prediction_file (str) – Path to saved predictions. Default: None

  • model_type (str) – only rnn available for the moment

Returns:

(pandas.DataFrame) holds the performance metrics for this dataframe

supernnova.validation.metrics.get_rnn_performance_metrics_singlemodel(settings, df, host_zspe_list)[source]

Compute performance metrics (accuracy, AUC, purity etc) for an RNN model

  • Compute metrics around peak light (i.e. PEAKMJD) and for the full lightcurve.

  • For bayesian models, compute multiple predictions per lightcurve and then take the median

Parameters:

  • settings (ExperimentSettings) – custom class to hold hyperparameters

  • df (pandas.DataFrame) – dataframe containing a model’s predictions

  • host_zspe_list (list) – available host galaxy spectroscopic redshifts

Returns:

(pandas.DataFrame) holds the performance metrics for this dataframe

supernnova.validation.metrics.get_uncertainty_metrics_singlemodel(df)[source]

For any lightcurve, compute the standard deviation of the model’s predictions (this is only valid for bayesian models which yield a distribution of predictions).

Then, compute the mean and std dev of this distribution across all lightcurves A higher mean indicates a model which is less confident in its predictions

Parameters:

df (pandas.DataFrame) – dataframe containing a model’s predictions

Returns:

(pandas.DataFrame) holds the uncertainty metrics for this dataframe

supernnova.validation.metrics.get_entropy_metrics_singlemodel(df, nb_classes)[source]

Compute the entropy of the predictions Low entropy indicates a model that is very confident of its predictions

Parameters:

  • df (pandas.DataFrame) – dataframe containing a model’s predictions

  • nb_classes (int) – the number of classes in the classification task

Returns:

(pandas.DataFrame) holds the entropy metrics for this dataframe

supernnova.validation.metrics.get_calibration_metrics_singlemodel(df)[source]

Compute probability calibration dataframe. If the calibration curve is close to identity, the model is considered well-calibrated.

Parameters:

df (pandas.DataFrame) – dataframe containing a model’s predictions

Returns:

(pandas.DataFrame) holds the calibration metrics for this dataframe

supernnova.validation.metrics.get_classification_stats_singlemodel(df, nb_classes)[source]

Find out how many lightcurves are classified in each class

Parameters:

  • df (pandas.DataFrame) – dataframe containing a model’s predictions

  • nb_classes (int) – the number of classes in the classification task

Returns:

(pandas.DataFrame) holds the calibration metrics for this dataframe