0
$\begingroup$

I want to build a neural network over a data set. My idea is to use cross-validation on a training set to select the "best" neural network (and evaluate it on a separate test set) and to use nested cross-validation to make some statistical predictions. I'd use nested CV to plot bias and variance of my grid search's hyper-parameters. This way I can estimate my method's performance.

If these assumptions are not wrong, what should I do first? Model selection or estimation?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

I think you misunderstood something about the nested cross validation: Hyperparameter tuning / model selection is not done before nor after nested cross validation, it is done in the inner loop of the nested cross validation.

Probably related to the misunderstanding: grid hyperparameters in themselves do not cause a bias in generalization error erstimation. The (optimistic) bias is caused by the selection of the (apparently) best hyperparameter set.

Improve this answer
$\endgroup$
10
  • $\begingroup$ Okay, I know that nested cross validation returns k models from the k inner loops (each one performs model selection). Have I to choose between one of these k models? $\endgroup$
    – Stefano Nardo
    Dec 19, 2016 at 17:39
  • 1
    $\begingroup$ You don't. Your outer cross validation estimates generalization error for a training function that includes tuning. Thus, your final model (trained on the whole data set is basically the inner CV loop applied to the whole data set. See e.g. stats.stackexchange.com/a/65156/4598, stats.stackexchange.com/a/233027/4598, stats.stackexchange.com/a/245169/4598 $\endgroup$
    – cbeleites unhappy with SX
    Dec 19, 2016 at 17:49
  • $\begingroup$ I can't see where I'm wrong. You say that the best result is when surrogate models are similar, that is when the same hyper-parameters win in the inner loop. This is what you called a stable method. If my method is stable I can use the method over the whole set. What if is unstable? Do I change it and try again? However, let's suppose it is stable. As you said in other question, I run f (whole data set). f is essentially the inner loop. But f is just training+validation, without a third set for testing purpose. Isn't not including test set wrong? $\endgroup$
    – Stefano Nardo
    Dec 19, 2016 at 18:33
  • $\begingroup$ Tell me if I'm wrong: my goal is to get a stable f (obviously a low error too) at the end of the nested CV. If this requirement is satisfied I can run f on the whole data set. $\endgroup$
    – Stefano Nardo
    Dec 19, 2016 at 19:08
  • $\begingroup$ I think you still have important misconceptions about what cross validation does, and why it is used. "But f is just training+validation, without a third set for testing purpose." This is the last step after the cross validation, it is not a replacement for cross validation. With the CV, you test f as a training procedure. You then use the CV results as approximation to the (unmeasured) performance of f (whole data set). This approximation (which is an extrapolation to a slightly larger training set) does not work if already the surrogate models have widely varying performance. $\endgroup$
    – cbeleites unhappy with SX
    Dec 20, 2016 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.