As we all know, depending solely on R squared to determine goodness of fit is faulty because the R squared value will always increase when more predictors are added, regardless of their physical correlation. With this in mind, many people recommend using the adjusted R squared value, especially if the equation will be used to estimate future values.
However, I recently stumbled upon something called the predicted R squared value, which relies on the predicted residual error sum of squares (PRESS) statistic. In this method,
- A data point from your dataset is removed
- A refitted linear regression model is generated
- The removed data point is plugged into the refitted linear model, generating a predicted value
- The removed data point is placed back into your dataset. Repeat from step 1 for the next data point until all data points have had a chance to be removed.
Afterward, you get a vector of predicted values which you subtract from their analogous true values to get a predicted residual value, which can be squared to convert it into a PRESS value. When calculating the predicted R Squared, the PRESS effectively replaces the sum of squares residuals value in the R Squared formula.
My question is which method is better at taking into account overmodeling, adjusted R Squared or predicted R Squared? The adjusted R Squared formula relies on the R squared value and the dataset size and predictor number, but the predicted R Squared completely re-calculates the sum of squares residual.
A part of a project I'm working on requires creating a multilinear regression model to correlate predictors (fictional example) such as grass length, intensity of green grass color, quantity of grass per square area, etc. (x variables) to cow happiness (y variable). The adjusted R squared and predicted R squared values react completely differently when I go from 3 to 4 to 5 predictors. The adjusted R squared value stays pretty much constant around 91% from 3 to 5 predictors. However, the predicted R squared value decreases from 87% to 71% to 60%. Which R Squared value applies more here?
Any help is appreciated. Thanks!