Does adjusted R-square seek to estimate fixed score or random score population r-squared?

Question

Population r-square $\rho^2$ can be defined assuming fixed scores or random scores:

• Fixed scores: The sample size and the particular values of the predictors are held fixed. Thus, $\rho^2_f$ is the proportion of variance explained in the outcome by the population regression equation when the predictor values are held constant.

• Random scores: The particular values of the predictors are drawn from a distribution. Thus, $\rho^2_r$ refers to the proportion of variance explained in the outcome in the population where the predictor values correspond to the population distribution of the predictors.

I've previously asked about whether this distinction makes much difference to estimates of $\rho^2$. I've also asked generally about how to calculate an unbiased estimate of $\rho^2$.

I can see that as the sample size gets larger the distinction between fixed-score and random-score gets less important. However, I'm trying to confirm whether adjusted $R^2$ is designed to estimate fixed score or random score $\rho^2$.

Questions

• Is adjusted $R^2$ designed to estimate fixed score or random score $\rho^2$?
• Is there a principled explanation of how the formula for adjusted r-square relates to one or other form of $\rho^2$?

Background to my confusion

When I read Yin and Fan (2001, p.206) they write:

One of the basic assumptions of the multiple regression model is that the values of the independent variables are known constants and are fixed by the researcher before the experiment. Only the dependent variable is free to vary from sample to sample. That regression model is called the fixed linear regression model.

However, in social and behavioral sciences, the values of independent variables are rarely fixed by the researchers and are also subject to random errors. Therefore, a second regression model for applications has been suggested, in which both dependent and independent variables are allowed to vary (Binder, 1959; Park & Dudycha, 1974). That model is called the random model (or correction model). Although the maximum likelihood estimates of the regression coefficients obtained from the random and fixed models are the same under normality assumptions, their distributions are very different. The random model is so complex that more research is needed before it can be accepted in place of the commonly used fixed linear regression model. Therefore, the fixed model is usually applied, even when the assumptions are not met completely (Claudy, 1978). Such applications of the fixed regression model with assumptions violated would cause "overfitting," because the random error introduced from the less-than-perfect sample data tends to be capitalized in the process. As a result, the sample multiple correlation coefficient obtained that way tends to overestimate the true population multiple correlation (Claudy, 1978; Cohen & Cohen, 1983; Cummings, 1982).

So I wasn't clear whether the above statement is saying that adjusted $R^2$ compensates for error introduced by the random model or whether this was just a caveat in the paper flagging the existence of the random model, but that the paper was going to focus on the fixed model.

References

• Yin, P., & Fan, X. (2001). Estimating $R^2$ shrinkage in multiple regression: A comparison of different analytical methods. The Journal of Experimental Education, 69(2), 203-224. PDF

Answer 1

Raju et al (1997) note that

Pedhazur (1982) and Mitchell & Klimoski (1986) have argued that results are
relatively unaffected by the model [fixed-x or random-x] selected when Ns are at least of moderate size (approximately 50).

Nonetheless, Raju et al (1997) classify some adjusted $R^2$ formulas for estimating $\rho^2$ as "Fixed X formulas" and "Random X formulas".

Fixed X formulas: Several formulas are mentioned including the formula proposed by Ezekiel (1930) which is standard in most statistical software:

$$\hat{\rho}_{(E)}^2 = 1 - \frac{N-1}{N-p-1}(1-R^2)$$

Thus, the short answer to the question is the standard adjusted $R^2$ formula typically reported and built into standard statistical software is an estimate of fixed-x $\rho^2$.

Random X formulas:

Olkin and Pratt (1958) proposed a formula

$$\hat{ \rho}^2 _{(OP)} = 1 - \left[ {\frac{{N - 3}}{{N - p - 1}}} \right](1 - {R^2})F\left[ {1,1;\frac{{N - p + 1}}{2};(1 - {R^2})} \right]$$ where F is the hypergeometric function.

Raju et al (1997) explain how various other formulas, such as Pratt's and Herzberg's "are approximations to the expected hypergeometric function". E.g., Pratt's formula is

$${\hat \rho}^2_{(P)} = 1 - \frac{{(N - 3)(1 - {R^2})}}{{N - p - 1}}\left[ {1 + \frac{{2(1 - {R^2})}}{{N - p - 2.3}}} \right]$$

How do estimates differ? Leach and Hansen (2003) report present a nice table showing the effect of different formulas on a sample of different published datasets in psychology (see Table 3). The mean Ezekiel $R^2_{adj}$ was .2864 compared to Olkin and Pratt $R^2_{adj}$ of .2917 and Pratt $R^2_{adj}$ of .2910. As per Raju et al's initial quotation about the distinction between fixed and random-x formulas being most relevant to small sample sizes, Leach and Hansen's table shows how the difference between Ezekiel's fixed-x formula and Olkin and Pratt's random-x formula is most prominent in small sample sizes, particularly those less than 50.

References

• Leach, L. F., & Henson, R. K. (2003). The use and impact of adjusted R2 effects in published regression research. In annual meeting of the Southwest Educational Research Assocation, San Antonio, TX. PDF
• Mitchell, T. W., & Klimoski, R. J. (1986). Estimating the validity of cross-validity estimation. Journal of Applied Psychology, 71, 311-317.
• Pedhazur, E. J. (1982). Multiple Regression in Behavioral Research (2nd ed.) New York: Holt, Rinehart, and Winston.
• Raju, N. S., Bilgic, R., Edwards, J. E., & Fleer, P. F. (1997). Methodology review: Estimation of population validity and cross-validity, and the use of equal weights in prediction. Applied Psychological Measurement, 21(4), 291-305.