Im assuming the model & estimations would be less accurate, causing the residuals to be larger, therefore, it makes R^2 larger. Just want to make sure and see if anyone has any insight for me. Thanks!
$\begingroup$
$\endgroup$
2
-
5$\begingroup$ $R^2$ will (essentially always) go down at least a little - but it may be so little as to make no discernable difference. $\endgroup$– Glen_bCommented Sep 30, 2014 at 7:16
-
1$\begingroup$ I wouldn't say that the estimators are less accurate. If you're trying to perform inference while adjusting for other related factors, adding a bunch of variables for no reason will give you "less accurate" (imprecise) measurement. Prediction is a different case. $\endgroup$– AdamOCommented Sep 30, 2014 at 21:22
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Removal of a variable from regression cannot increase R squared because adding a new variable cannot decrease residual sum of squares (R squared = 1 - residual sum of squares/total sum of squares). But it doesn't mean that you should add in your model as mny variable as possible. In order to determine the effecteveness of added variable use adjusted R squared or information criteria (Akaike's or Schwarz's).
$\begingroup$
$\endgroup$
Taking out a variable will remove some of the "wiggle room" for the model to fit the data, so yes, the fitted points probably won't be as close to the data, so the $R^2$ will probably be lower.
However, keep in mind that this does not necessarily mean that the model or estimates are less accurate because you took out a variable. That variable could have been completely meaningless, but the $R^2$ could still be higher because adding an extra variable gives your model another opportunity to overfit.
$\begingroup$
$\endgroup$
5
The answer to your question is context dependent. Just about anything can happen
1) The trivial case is the quality of the fit reducing. This happens, when you remove a variable which explains part of the variance not explained by any other variable (you have written "makes $R^{2}$ larger which is not correct.. Recall $R^{2}$ = $ 1 - \frac {S_{Res}}{S_{tot}}$ so when $S_{Res}$ increases, $R^{2}$ will decrease).
2) The quality of the fit remains the same: Consider the case where you are fitting a regression using multiple variables and you decide to remove a variable X which is perfectly correlated with another variable Y (which you do not remove). In this case, you would expect the quality of the fit to remain the same.
3) Suppose we have 100 observations and 300 variables we are using to predict. You do not have enough predictive power in this case BUT if you know before hand (say) 295 are useless and remove them your $R^{2}$ may increase
-
$\begingroup$ Thanks for the answer, that makes perfect sense. I was mixing up the disturbance term with R square, which was dumb. I was also making the false assumption that every input included is beneficial to your estimations. Would you say that is where I went wrong? What you are saying essentially is that it depends on if the input you are removing has a smaller or larger, u term, than on average to know how R squared would change? $\endgroup$ Commented Sep 30, 2014 at 5:24
-
$\begingroup$ Cases 2 and 3 seem equivalent: they concern situations of perfect multicollinearity. But then the software, in order to obtain estimates at all, necessarily deletes some redundant variables anyway. Only case 1 can actually occur. $\endgroup$– whuber ♦Commented Sep 30, 2014 at 6:29
-
$\begingroup$ 3 need not necessarily be a case of collinearity. I should probably have been more clear. Suppose 295 are just noise variables ( which I happen to know before hand), removing them might improve the quality of the fit. When all 300 are maintained, some "true" signal variables may need to be sacrificed in lieu of multiple testing stringencies. It is a common problem in high dimensions. This happens because we just cannot say enough from the data we have. $\endgroup$– SidCommented Sep 30, 2014 at 15:03
-
$\begingroup$ I agree 2 is a trivial case but there is nothing preventing it from being part of a regression setting. $\endgroup$– SidCommented Sep 30, 2014 at 15:08
-
3$\begingroup$ The only possible direction of change of $R^2$ is down when variables are removed: that's a purely mathematical theorem. It simply is not possible for $R^2$ to increase in case (3). If it stays the same, that means the removed variables were linearly dependent on the ones that were kept in. $\endgroup$– whuber ♦Commented Sep 30, 2014 at 16:39