Why would my cross validation accuracy be noticeably lower than my actual testing accuracy?

Question

I coded up a nested cross validation scheme for model selection, feature selection, and hyper parameter optimization. Here are some results I got:

Model selection accuracies (for the two models I tested): 0.65 0.66

Model hyperparameter loop accuracies (for testing the hyperparameters of the winning model): 0.65 0.62 0.63 0.62

Final Model accuracy: 0.721519

The final model accuracy is higher than the cross validation results. Usually this is a good thing, but the difference seems a little too large for comfort. Assuming I coded this correctly, I can only attribute lower accuracy to a smaller sample size in the cv phase (FYI: I used a .8 split for any train and test split including the inner and outer nested validation splits).

Is there anything that comes to mind for why else this might happen?

Answer 1

Some back-of-the-envelope ideas:

• First thought is always a quick glance at uncertainty due to number of tested cases: with n = 800 and observed accuracy .721519, the 95 % confidence interval is ca. 69 - 75 %, and
  for n = 640 and observecd accuracy .63, the 95 % confidence interval is ca. 59 - 67 %.
  Not overlapping, but also not that far apart. In practice, random error may be larger if the models are not totally stable.

• Second thought: some popular choices of how to actually select the optimal model are biased towards too complex models, i.e. do systematically overfit. Thus, generalization error may be lower than optimal. OTOH, a popular version of how to obtain the final model is to take the hyperparameters found in the optimization and use them for fitting on the larger (outer cross validation) training data (as opposed to running another optimization on the larger training set and take those hyperparameters). This may counteract to a certain extent the overfitting inside the optimization, which could lead to better models. In other words, you may be lucky in having a sweet spot where overfitting and underfitting roughly cancel out each other.

---

(I'd formulate choice of model, features and model hyperparameters all as a single hyperparameter optimization problem, thus 1 nesting level)