When performing model selection, suppose you wanted to select the number of hidden layers for your deep net. That would be too expensive to perform cross-validation on the training set to determine which model is the best using a paired t-test on the k-folds.
You would then do a split of your dataset into a training/validation/test set. You then train your deep net with varying numbers of hidden layers and computer you error on the validation set. Andrew Ng said you would then pick the model with the lowest error on the validation set.  However, would it make sense to perform some type of statistical test to determine if the difference was significant? Like for instance a paired t-test by pairing up the the examples in the validation set?
I'm thinking the absolute lowest error might not be the best measure because if two models weren't statistically significantly different then you could just pick the one with the least number of hidden layers.