I coded up a nested cross validation scheme for model selection, feature selection, and hyper parameter optimization. Here are some results I got:
Model selection accuracies (for the two models I tested): 0.65 0.66
Model hyperparameter loop accuracies (for testing the hyperparameters of the winning model): 0.65 0.62 0.63 0.62
Final Model accuracy: 0.721519
The final model accuracy is higher than the cross validation results. Usually this is a good thing, but the difference seems a little too large for comfort. Assuming I coded this correctly, I can only attribute lower accuracy to a smaller sample size in the cv phase (FYI: I used a .8 split for any train and test split including the inner and outer nested validation splits).
Is there anything that comes to mind for why else this might happen?