I am comparing linear regression models.
For the first measure, I compute the aic, adjusted r-squared and standard-error-of-the-regression with the average squared error computed on the validation fold (currently numFolds=10, N=17,000 records) (by training on the remaining folds).
For the second measure, I simply compute the cross-validated squared error and absolute mean error.
The aic, adjusted $R^2$ and standard-error-of-the-regression all roughly agree that the model should contain 7 terms whereas the cross-validate error measures seem to prefer 22 terms.
Looking at the $R^2$ curve as a function of terms, I would agree with the aic/adjusted $R^2$. The $R^2$ flattens out; terms 8 through 22 seem superfluous.
This CMU lecture seems to prefer cross-validated error: http://www.stat.cmu.edu/~larry/=stat705/Lecture16.pdf however, this would seem to overfit.