I understand adjusted r-squared and AIC can be used to select an ideal model from a group. Higher AIC is worse but higher ar2 is better.

After adding a categorical variable to an OLS model, my adjusted r-squared went up and so did my AIC. Some of the categorical values are associated with a high p-value.

Three questions:

  1. How can I interpret the discrepancy?
  2. I tend to favor including the variable because of the ar2 effect. Is this a fair justification?
  3. Is either metric generally preferred among AR2 and AIC?
  • $\begingroup$ Why are you choosing between these models? Why don't you just have one model that you fit, and that's the model? $\endgroup$
    – Dave
    Oct 13 '21 at 17:29
  • 1
    $\begingroup$ See this thread regarding the lack of justification for $R^2_{adj.}$ as a model selection criterion. $\endgroup$ Oct 13 '21 at 17:32
  • $\begingroup$ Thanks @RichardHardy! @Dave my use case is identifying the most important features (eg forward selection/backwards elimination) $\endgroup$ Oct 13 '21 at 17:53
  • $\begingroup$ Stepwise model selection is problematic. // Gelman agrees. $\endgroup$
    – Dave
    Oct 13 '21 at 17:57
  • $\begingroup$ @Dave I have theory-based supervision of the variable pool, I'm not maximizing r-squared, I am performing cross-validation...I think those objections are largely leveraged against some other form of selection. Do you have anything in the way of positive, preferred suggestion? $\endgroup$ Oct 13 '21 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.