Disagreement of k-fold cross-validated error with aic, adjusted r-squared etc

I am comparing linear regression models.

For the first measure, I compute the aic, adjusted r-squared and standard-error-of-the-regression with the average squared error computed on the validation fold (currently numFolds=10, N=17,000 records) (by training on the remaining folds).

For the second measure, I simply compute the cross-validated squared error and absolute mean error.

The aic, adjusted $R^2$ and standard-error-of-the-regression all roughly agree that the model should contain 7 terms whereas the cross-validate error measures seem to prefer 22 terms.

Looking at the $R^2$ curve as a function of terms, I would agree with the aic/adjusted $R^2$. The $R^2$ flattens out; terms 8 through 22 seem superfluous.

This CMU lecture seems to prefer cross-validated error: http://www.stat.cmu.edu/~larry/=stat705/Lecture16.pdf however, this would seem to overfit.

• Why do you think a cross-validation procedure might lead to overfitting? That seems impossible by definition if implemented correctly. – mkt Nov 30 '17 at 12:13
• A key fact about model comparisons that is often ignored is that the data used cannot be randomly generated for each model. In other words, a seed must be used to ensure that the partitions between train and test, e.g., the k-folds, are exactly the same for each model. This is a necessary precondition for any between-model comparisons to be accurate. In the absence of this, there is no way to be sure that you have an apples to apples result and comparison. – DJohnson Nov 30 '17 at 12:16
• Thank you both. Perhaps you are right that I did not execute the n-fold procedure correctly or that it's just a bug - I will double check this and report back. However I am not using replacement to select folds. There is a random field between 0 and 99 stored with each record. For each training, a test slice is reserved where the random field is between e.g., 0-9, 10-19 etc. and the rest used to train the model e.g., 10-99, {0-9 & 20-90} etc. – dreamer Nov 30 '17 at 19:22
• @mkt, Cross-validation by itself does not protect against over-fitting because some redundancy might be left between the different partitions. – Itamar Dec 5 '17 at 7:15
• @Itamar Thanks, I was not aware of this. Could you point me towards resources that explain this further? – mkt Dec 5 '17 at 15:57