I would like to figure out which hyperparameters/network architectures work best on a specific binary dataset. To compare them, I do k-fold cross-validation, use different MLP or ConvNet architectures and tune hyperparameters for both. For each, I compute the ROC-curves and am supposing that the iterations with higher AUC (area under curve) are necessarily better. Is this justifiable?
2
-
$\begingroup$ This is 100% justifiable in ML. $\endgroup$– SmallChessCommented Apr 27, 2017 at 6:07
-
$\begingroup$ thanks, are there other metrics worth exploring as well? $\endgroup$– jerpintCommented Apr 27, 2017 at 16:27
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
This is standard practice: run a lot of experiments, pick the one with the highest score on the hold-out. The caveats here are straightforward: The AUC is a statistic, and like every other statistic it is subject to random variation, so picking the highest AUC value might be slightly worse in reality than another hyper-parameter configuration by chance. But this is the nature of living with randomness.
You might be interested in this answer, which indirectly addresses how one might efficiently search for a good hyper-parameter configuration: Optimization when Cost Function Slow to Evaluate
You could also carry out a statistical test to compare whether one ROC curve is significantly better than another. See: Statistical significance (p-value) for comparing two classifiers with respect to (mean) ROC AUC, sensitivity and specificity
There are all sorts of other metrics that people use to measure model performance. Some of them are:
log loss/cross-entropy
Brier score
TPR at a specific FPR
Area under the precision-recall curve
Generally, you might be interested in scoring-rules.
