I would like to figure out which hyperparameters/network architectures work best on a specific binary dataset. To compare them, I do k-fold cross-validation, use different MLP or ConvNet architectures and tune hyperparameters for both. For each, I compute the ROC-curves and am supposing that the iterations with higher AUC (area under curve) are necessarily better. Is this justifiable?

  • $\begingroup$ This is 100% justifiable in ML. $\endgroup$
    – SmallChess
    Apr 27 '17 at 6:07
  • $\begingroup$ thanks, are there other metrics worth exploring as well? $\endgroup$
    – jerpint
    Apr 27 '17 at 16:27

This is standard practice: run a lot of experiments, pick the one with the highest score on the hold-out. The caveats here are straightforward: The AUC is a statistic, and like every other statistic it is subject to random variation, so picking the highest AUC value might be slightly worse in reality than another hyper-parameter configuration by chance. But this is the nature of living with randomness.

You might be interested in this answer, which indirectly addresses how one might efficiently search for a good hyper-parameter configuration: Optimization when Cost Function Slow to Evaluate

You could also carry out a statistical test to compare whether one ROC curve is significantly better than another. See: Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity

There are all sorts of other metrics that people use to measure model performance. Some of them are:

  • log loss/cross-entropy

  • Brier score

  • TPR at a specific FPR

  • Area under the precision-recall curve

Generally, you might be interested in .


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.