When performing model selection, suppose you wanted to select the number of hidden layers for your deep net. That would be too expensive to perform cross-validation on the training set to determine which model is the best using a paired t-test on the k-folds.

You would then do a split of your dataset into a training/validation/test set. You then train your deep net with varying numbers of hidden layers and computer you error on the validation set. Andrew Ng said you would then pick the model with the lowest error on the validation set. [1] However, would it make sense to perform some type of statistical test to determine if the difference was significant? Like for instance a paired t-test by pairing up the the examples in the validation set?

I'm thinking the absolute lowest error might not be the best measure because if two models weren't statistically significantly different then you could just pick the one with the least number of hidden layers.

[1] https://youtu.be/z6aBwtEby_Y?t=10m27s


1 Answer 1


This whole process is involves subtle problems. Unless your signal:noise ratio is extremely high, the model derived from a training sample is likely to have some instability, so the process you are using is essentially validating an "example" model. A unified resampling approach (bootstrapping is recommended) validates the process used to find a model, recognizes the uncertainties from model selection, and in the process also provides a darn good estimate of the likely future performance of the "final" model. More thoughts are at http://www.fharrell.com/2017/01/split-sample-model-validation.html .


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.