I'm inclined to think that the new variable is not correlated to the response. But could the new variable be correlated to another variable in the model?
7
-
$\begingroup$ It depends, could you provide us with some reduced data lines or output from your linear models. Without more information it's hard to assist you $\endgroup$– OliverFishCodeMar 7, 2019 at 19:54
-
5$\begingroup$ It shouldn't stay exactly the same unless it is perfectly orthogonal to your response, or is a linear combination of the variables already included. It may be that the change is smaller than the number of decimal places displayed. $\endgroup$– gung - Reinstate MonicaMar 7, 2019 at 20:02
-
6$\begingroup$ @gung What you can infer is that the new variable is orthogonal to the response modulo the subspace generated by the other variables. That's more general than the two options you mention. $\endgroup$– whuber ♦Mar 7, 2019 at 20:12
-
$\begingroup$ @whuber, yes, I suppose so. $\endgroup$– gung - Reinstate MonicaMar 7, 2019 at 20:17
-
$\begingroup$ Test your variables for multicollinearity en.wikipedia.org/wiki/Multicollinearity probably some features are linearly connected. Use caret package and vif() in R sthda.com/english/articles/39-regression-model-diagnostics/… $\endgroup$– Fierce82Mar 7, 2019 at 22:33
|
Show 2 more comments
2 Answers
$\begingroup$
$\endgroup$
Seeing little to no change in $R^2$ when you add a variable to a linear model means that the variable has little to no additional explanatory power to the response over what is already in your model. As you note, this can be either because it tells you almost nothing about the response or it explains the same variation in the response as the variables already in the model.
answered Mar 7, 2019 at 19:55
$\begingroup$
$\endgroup$
1
As others have alluded, seeing no change in $R^2$ when you add a variable to your regression is unusual. In finite samples, this should only happen when your new variable is a linear combination of variables already present. In this case, most standard regression routines simply exclude that variable from the regression, and your $R^2$ will remain unchanged because the model was effectively unchanged.
As you notice, this does not mean the variable is unimportant, but rather that you are unable to distinguish its effect from that of the other variables in your model.
More broadly however, I (and many here at Cross Validated) would caution against using R^2 for model selection and interpretation. What I've discussed above is how the $R^2$ could not change and the variable still be important. Worse yet, the $R^2$ could change somewhat (or even dramatically) when you include an irrelevant variable. Broadly, using $R^2$ for model selection fell out of favor in the 70s, when it was dropped in favor of AIC (and its contemporaries). Today -- a typical statistician would recommend using cross validation (see the site name) for your model selection.
In general, adding a variable increases $R^2$ -- so using $R^2$ to determine a variables importance is a bit of a wild goose chase. Even when trying to understand simple situations you will end up with a completely absurd collection of variables.
-
$\begingroup$ Could you elaborate on the $R^2$ could change somewhat (or even dramatically) when you include an irrelevant variable, specifically on the case of a dramatical change? In which sense would the variable then be irrelevant? $\endgroup$ Mar 7, 2019 at 21:38