Most likely sources of divergence between (adjusted)-R squared and out-of-sample predictive performance

Question

I'm wondering which invalid assumptions are most likely to explain the wild discrepancies between a model's R-squared as a measure of predictive performance, and the actual out-of-sample predictive accuracy of a model.

Although it is known that in the standard OLS case the R-squared criterion, $R^2 = 1 - \frac{\text{SS}_\text{reg}}{\text{SS}_\text{tot}}$ - the percentage of variance explained by a model as measured by the remaining sum of squares in the model relative to the total sum of squares in the data - tends to over-estimate the predictive power of linear models since it does not address the process of fitting the model in terms of the degrees of freedom used to model the data.

The adjusted R-squared criterion, $R^2 = 1 - \frac{\text{SS}_\text{reg}}{\text{SS}_\text{tot}} \frac{n - 1}{n - p - 1}$ scales the standard R-squared downwards to address this issue (similar to the scaling of the estimate for the population variance), with $p$ the number of coefficients estimated and thus the last term being less than 1.

However, when testing the accurateness of the $R^2$ in real-world applications as an estimate of predictive power, it often seems to be the case that the adjusted R-squared (and thus also the normal R-squared) grossly over-estimates the predictive performance of a model in terms of out-of-sample prediction (e.g. using a cross-validation or test/train split on the data).

Intuitively, this will have to do with the invalidity of the assumptions of linearity and additivity which are usually present in linear regressions but I'm wondering what the largest sources of discrepancy would be?

Answer 1

The adjusted $R^2$ is actually an asymptotically unbiased estimate of the population $R^2$, at least according to Wikipedia. So if you use cross-validation of straightforward linear regression, the cross-validated $R^2$ shouldn't be much different from the adjusted $R^2$.

Reasons why you are seeing overfitting may be because you include model/variable selection in the cross-validation? In this case, your adjusted $R^2$ is invalid cos it doesn't take into account of the model/variable selection.

If you are talking about real out-of-sample prediction, however, (not cross-validation based on random splitting), then typically the overfitting is more severe, because the true coefficients $b$ may not even be the same in different samples.

Answer 2

As far as I can see, your question is why your model works better in-sample than out-of-sample.

This is a standard symptom of overfitting. You have estimated a model, which maximises your R2 in-sample. When maximising in-sample fit, you are accidentally capturing some of the residual variance (~ 'noise' = the stuff that should be in your error term) in your estimated regression coefficients.

To get an intuition why this happens, remember that when we prove that OLS is an optimal linear estimator for y, we start by assuming that y = Xb + e, where X is a known design matrix. In the real world, we typically employ either various heuristics or formal model selection criteria to choose what we put into X.

In theory, even a linear-in-parameters model such as y = Xb+e can generate fairly complicated nonlinearities via things such as polynomials and splines. The problem is just that you need to know the non-linearities in advance.

I have not seen a good discussion on which of the standard OLS assumptions are broken most often. I would expect that to vary a lot case-by-case. I would try to assess this graphically by fitting (e.g.) local-linear and linear models and comparing their fits graphically.