1
$\begingroup$

I coded up a nested cross validation scheme for model selection, feature selection, and hyper parameter optimization. Here are some results I got:

Model selection accuracies (for the two models I tested): 0.65 0.66

Model hyperparameter loop accuracies (for testing the hyperparameters of the winning model): 0.65 0.62 0.63 0.62

Final Model accuracy: 0.721519

The final model accuracy is higher than the cross validation results. Usually this is a good thing, but the difference seems a little too large for comfort. Assuming I coded this correctly, I can only attribute lower accuracy to a smaller sample size in the cv phase (FYI: I used a .8 split for any train and test split including the inner and outer nested validation splits).

Is there anything that comes to mind for why else this might happen?

$\endgroup$
  • $\begingroup$ What is your sample size? $\endgroup$ – cbeleites Jul 5 '18 at 14:57
  • $\begingroup$ ~800. this is then split into train/test or validation by a .8/.2 ratio at each nesting step $\endgroup$ – skim Jul 5 '18 at 16:07
  • $\begingroup$ I think that with "just" ~800 samples CV is a bit too restrictive (especially given that we are not talking about repeated-CV), I do not think you will get very reliable estimates to begin with. I would strongly suggest using bootstrap in this case. $\endgroup$ – usεr11852 Jul 5 '18 at 17:40
1
$\begingroup$

Some back-of-the-envelope ideas:

  • First thought is always a quick glance at uncertainty due to number of tested cases: with n = 800 and observed accuracy .721519, the 95 % confidence interval is ca. 69 - 75 %, and
    for n = 640 and observecd accuracy .63, the 95 % confidence interval is ca. 59 - 67 %.
    Not overlapping, but also not that far apart. In practice, random error may be larger if the models are not totally stable.

  • Second thought: some popular choices of how to actually select the optimal model are biased towards too complex models, i.e. do systematically overfit. Thus, generalization error may be lower than optimal. OTOH, a popular version of how to obtain the final model is to take the hyperparameters found in the optimization and use them for fitting on the larger (outer cross validation) training data (as opposed to running another optimization on the larger training set and take those hyperparameters). This may counteract to a certain extent the overfitting inside the optimization, which could lead to better models. In other words, you may be lucky in having a sweet spot where overfitting and underfitting roughly cancel out each other.


(I'd formulate choice of model, features and model hyperparameters all as a single hyperparameter optimization problem, thus 1 nesting level)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.