1
$\begingroup$

How would I go about tuning my neural network hyperparameters, getting an estimate of its performance on unseen data, while finally training on the entire dataset? The only way I can think of is maybe using nested cross validation but for a neural network that can be unfeasible to do computation wise. Is there any other method?

$\endgroup$
1
$\begingroup$

Welcome to CV!

If your test data influences the parameters in any way, you can't then claim to have a (somewhat) independent performance estimate. Similarly, if you were to train on the test data after calculating the performance on it, then you no longer know whether this estimate is representative of the new performance. Also, how would you assess overfitting past that point?

Alternatives to partitioning your data include optimism bootstrap, although this is no less computationally costly. If your data are really limited and the computational cost is too high to repeat the fitting process, perhaps you should be looking into less complex models that can be fitted faster.

$\endgroup$
  • $\begingroup$ So to paraphrase, if I want to select my best model and obtain a measure of its production performance I must have a separate test set completely untouched by the training. In the first answer to this post here they suggest using nested CV for model selection and performance estimation on an entire dataset training scheme. If I don't want to use nested CV, then I must have a entirely separate testing set? $\endgroup$ – pancakes77 Apr 30 '19 at 1:10
  • $\begingroup$ Yes, exactly. Either that or a nested cross-validation scheme. $\endgroup$ – Frans Rodenburg Apr 30 '19 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.