Relative newbie with quantitative analysis here, so forgive me if the question is naive or ill-specified. I have argued in a manuscript that along with using Beta values in linear multiple regression output to show the relative strength/magnitude of the influence of each independent variable on variation in the dependent variable, one can also remove independent variables separately (or as a category) note the reduction in adjusted r-squared values to compare the relative strength of each variable (or category). A reviewer of the manuscript questions if this last move is legitimate. I thought I read somewhere some months ago that this was a valid procedure, and it actually seemed like common sense to me, so I didn't bother to note the source. Now I can't find the source. Am I wrong in using change in adjusted r-squared when variables are removed from the regression model to evaluate the relative strength of variables?
-
1$\begingroup$ No this is not appropriate. R-squared increases just by increasing the number of variables, and vice versa. $\endgroup$– user2974951Sep 20, 2019 at 8:45
-
$\begingroup$ The OP stated "adjusted R2", in other words, it would not improve merely by the addition of variables. $\endgroup$– Paul HewsonSep 20, 2019 at 13:34
-
$\begingroup$ You use "relative strength" in two completely different senses: effect size (size of estimated parameter) and statistical significance (as measured by reduction in adjusted R-squared). In light of this contradiction, could you clarify for us just what you mean by "relative strength"? $\endgroup$– whuber ♦Sep 20, 2019 at 14:18
-
$\begingroup$ @whuber Maybe I misunderstand, but I thought reduction in adjusted r-squared is not directly connected with statistical significance. I thought both Beta values and adjusted r-squared are more directly connected with effect size. $\endgroup$– EthanSep 21, 2019 at 0:12
1 Answer
I think in principle, using adjusted R-squared is more sensible than using standard errors of individual predictors. However 1, people tend to use a likelihood based metric such as AIC for variable selection, which you try to minimise in a variable search analagous to maximising adjusted R-squared. However 2 this is a massive topic in statistical research. The risk you run with any variable selection method is overfitting, that is to say, finding a combination of predictor variables in your dataset which works brilliantly well, but a model that doesn't work well if applied to a new dataset on the same phenomenon. Here is one source on variable selection methods that might explain it better than I can: https://towardsdatascience.com/stopping-stepwise-why-stepwise-selection-is-bad-and-what-you-should-use-instead-90818b3f52df
-
$\begingroup$ @ Paul Hewson I very much appreciate this valuable answer. But variable selection and model fitting is not my most immediate problem. Let me give more detail. In my demographic research, I utilize many dummy variables for individual states and individual ethnicities as independent variables. Beta values are individual values related to each state or each ethnicity but Beta doesn't seem to help me discuss the relative role of states vs ethnicities (as categories; in magnitude of effect, not statistical significance). If adjusted r-squared cannot be used to compare those categories, what can? $\endgroup$– EthanSep 21, 2019 at 0:42
-
$\begingroup$ OK. I've definitely seen people compare AIC of models with (something akin to) states/ethnicities. It would tell you that a model fitted better with states than without, or with ethnicities than without. And whether a model with both fitted better than a model with just one. That's a method of variable selection, but a report of the search path would tell you how much the fit improved. AIC is very similar to adjusted R-squared except that it has a bias and variance correction. I don't think it's a million miles from what you are trying to do. $\endgroup$ Sep 23, 2019 at 8:35