7
$\begingroup$

In a recent thread, use of adjusted $R^2$ ($R^2_{adj.}$) is mentioned in the context of model selection, e.g.

The adjustment was invented as a solution to problems caused by variable selection

Question: Is there any justification for using $R^2_{adj.}$ for model selection? That is, does $R^2_{adj.}$ have any optimality properties in the context of model selection?

For example, AIC is an efficient criterion and BIC is a consistent one, but $R^2$ does not coincide with any of them and so makes me wonder if it can be optimal in any other sense.

$\endgroup$
6
1
$\begingroup$

I don't know if $R^2_{\text{adj.}}$ have any optimal properties for model selection, but it is surely taught (or at least mentioned) in that context. One reason might be because most students have met $R^2$ early on, so there is then something to build on.

One example is the following exam paper from University of Oslo (see problem 1.) The text used in that course, Regression Methods in Biostatistics Linear, Logistic, Survival, and Repeated Measures Models Second edition by Eric Vittinghoff, David V. Glidden, Stephen C. Shiboski and Charles E. McCulloch mentions $R^2_{\text{adj.}}$ early on in their chapter 10 on variable selection (as penalizing less than AIC, for example) but neither it nor AIC is mentioned in their summary/recommendations 10.5.

So it is maybe mostly used didactically, as an introduction to the problems of model selection, and not because of any optimality properties.

$\endgroup$
1
  • $\begingroup$ Thank you. The relevant part for me is the first part of your first sentence. $\endgroup$ Jun 1 '19 at 8:20
0
$\begingroup$

Answer for part1:

  1. If you add more variables, even totally insignificant variable, R2 can only go up. this is not the case with adjusted R2. You can try running multiple regression and then add random variable and see what happened to R2 and what happened to the adjusted R2.
$\endgroup$
14
  • 1
    $\begingroup$ Thanks! I do not think this constitutes a justification for the use of $R^2_{adj.}$ for model selection. It only justifies the preference of $R^2_{adj.}$ to $R^2$ when estimating the population $R^2$ under a very restrictive assumption, as I explain in the comment in the linked thread. I have edited my post to make my question clearer. $\endgroup$ May 14 '19 at 11:36
  • $\begingroup$ The adjusted R squared increases only if the under investigation variable improves the model more than just a random variable. $\endgroup$ May 15 '19 at 7:41
  • $\begingroup$ Thanks once more. Could you be more precise? In which sense exactly does it improve the model? Does the model become better at prediction under a given loss function? Or does a higher $R^2_{adj.}$ suggests the model is more likely to be the true model among a set of candidates? Or does it have any other properties that could justify it as a criterion for model selection? $\endgroup$ May 15 '19 at 7:57
  • $\begingroup$ R squared is the ratio of the explained variance and the adjusted R squared is less bias estimator for the population R squared. So you want higher explained variance in your model $\endgroup$ May 15 '19 at 8:12
  • $\begingroup$ The explained variance is the variance that the model knows to calculate. The not explain variance is the noise that the model can't predict, $\endgroup$ May 15 '19 at 12:50
0
$\begingroup$

I would propose six optimality properties.

  1. Overfit Mitigation
  2. Simplicity and Parsimony
  3. General Shared Understanding
  4. Semi-Efficient Factor Identification
  5. Robustness to Sample Size Change
  6. Explanatory Utility

Overfit Mitigation

What kind of model is overfit? In part, this depends on the model's use case. Suppose we are using a model to test whether a hypothesized factor-level relationship exists. In that case a model which tends to allow spurious relations is overfit.

"The use of an adjusted R2...is an attempt to account for the phenomenon of the R2 automatically and spuriously increasing when extra explanatory variables are added to the model." Wikipedia.

Simplicity and Parsimony

Parsimony is valued on normative and economic rationale. Occam's Razor is an example of a norm, and depending on what we mean by "justification," it might pass or fail.

The economic rationale for simplicity and parsimony are harder to dismiss:

  1. Complex models with many factors are expensive to gather data for.
  2. Complex models can be more expensive to execute.
  3. Complex models are hard to communicate and think through. Business and legal risks can result from this, as well as plain time spent communicating from one person to another.

Given two models with equal explanatory power (R2), then, AR2 selects for the simpler and more parsimonious model.

General Shared Understanding

Justification involves shared understanding. Consider a peer-review situation. If the reviewer and the reviewed lack a shared understanding of model selection, questions or rejections may occur.

R2 is an elementary statistical concept and those only familiar with elementary statistics still generally understand that R2 is gameable and AR2 is preferred to R2 for the above reasons.

Sure, there may be better choices compared to AR2 such as AIC and BIC, but if the reviewer is unfamiliar with these then their use may not succeed as a justification. What's worse, the reviewer may have a misunderstanding themselves and required AIC or BIC when they aren't required - that itself is unjustified.

My limited understanding indicates that AIC is now considered rather arbitrary by many - specifically the 2s in the formula. WAIC, DIC, and LOO-CV have been suggested as preferred, see here.

I hope by "justified" we don't mean "no better parameter exists" because it seems to me that some better parameter might always exist unbeknownst to us, therefore this style of justification always fails. Instead "justified" ought to mean "satisfies the requirement at hand" in my view.

Semi-Efficient Factor Identification

Caveat: I made up this term and I could be using it wrong :)

Basically, if we are interested in identifying true factor relations, we should expect p < 0.5, ie P(B) > P'(B). AR2 maximization satisfies this as adding a factor with p >= 0.5 will reduce AR2. Now this isn't an exact match because I think AR2 generally penalizes p > 0.35-ish.

It's true AIC penalizes more in general but I'm not sure that's a good thing if the goal is to identify all observed features that have an identifiable relation, say at least directionally, in a given data set.

Robustness to Sample Size Change

In the comments of this post, Scortchi - Reinstate Monica notes that it "makes no sense to compare likelihoods (or therefore AICs) of models fitted on different nos observations." In contrast, r-squared and adjusted r-squared are absolute measures that can be compared with a change in the number of samples.

This might be useful in the case of a questionnaire that includes some optional questions and partial responses. It's of course important to be mindful of issues like response bias in such cases.

Explanatory Utility

Here, we are told that "R2 and AIC are answering two different questions...R2 is saying something to the effect of how well your model explains the observed data...AIC, on the other hand, is trying to explain how well the model will predict on new data."

So if the use case is non-predictive, such as in the case of theory-driven, factor-level hypothesis testing, AIC may be considered inappropriate.

$\endgroup$
9
  • 1
    $\begingroup$ Note that under some assumptions, LOOCV is asymptotically equivalent to AIC. $\endgroup$ Oct 13 at 19:30
  • $\begingroup$ When you say "factor relations", what exactly do you mean? $\endgroup$ Oct 13 at 19:31
  • $\begingroup$ Here I am trying to generally express coefficient hypothesis testing. Usually I care about direction of effect, significance, and importance. This general term leaves room to explore an independent factor, lags, interactions, marginal effects, and other computed, derived, or engineered features. Possibly including causal testing. I am mainly thinking about OLS at the moment. $\endgroup$ Oct 13 at 20:40
  • $\begingroup$ Sometimes I’m also interested in how dependent variables relate to each other as a secondary concern. Eg does X2 “partial out” X1 $\endgroup$ Oct 13 at 20:41
  • $\begingroup$ I cannot find the relevant thread anymore to post this comment to but I will do it here. Just wanted to let you know I watched Ben Lambert's YouTube video about AIC, DIC, WAIC and LOOCV and did not find it convincing enough to drop AIC in favor of DIC or WAIC. Statements in the video (e.g. about popularity or approximation quality of AIC) are too general to be correct even if they may hold in special cases (though I understand this is intentional due to the format of the lecture series). DIC and WAIC may function OK for Bayesian modeling but I do not see if they are applicable outside of that. $\endgroup$ Oct 16 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.