# Model selection: before or after nested cross-validation?

I want to build a neural network over a data set. My idea is to use cross-validation on a training set to select the "best" neural network (and evaluate it on a separate test set) and to use nested cross-validation to make some statistical predictions. I'd use nested CV to plot bias and variance of my grid search's hyper-parameters. This way I can estimate my method's performance.

If these assumptions are not wrong, what should I do first? Model selection or estimation?

• I can't see where I'm wrong. You say that the best result is when surrogate models are similar, that is when the same hyper-parameters win in the inner loop. This is what you called a stable method. If my method is stable I can use the method over the whole set. What if is unstable? Do I change it and try again? However, let's suppose it is stable. As you said in other question, I run f (whole data set). f is essentially the inner loop. But f is just training+validation, without a third set for testing purpose. Isn't not including test set wrong? – Stefano Nardo Dec 19 '16 at 18:33
• Tell me if I'm wrong: my goal is to get a stable f (obviously a low error too) at the end of the nested CV. If this requirement is satisfied I can run f on the whole data set. – Stefano Nardo Dec 19 '16 at 19:08
• I think you still have important misconceptions about what cross validation does, and why it is used. "But f is just training+validation, without a third set for testing purpose." This is the last step after the cross validation, it is not a replacement for cross validation. With the CV, you test f as a training procedure. You then use the CV results as approximation to the (unmeasured) performance of f (whole data set). This approximation (which is an extrapolation to a slightly larger training set) does not work if already the surrogate models have widely varying performance. – cbeleites unhappy with SX Dec 20 '16 at 10:16