4
$\begingroup$

What does it mean when your coefficients in linear regression change, after you removed some variables?

I already checked the assumptions of linear regression. They seem to hold.

$\endgroup$
  • 4
    $\begingroup$ They are expected to change when you add/remove variables. If they change a lot, that means there might be confounding between the variable you removed and another variable. $\endgroup$ – Demetri Pananos Jan 2 at 22:45
2
$\begingroup$

Regression coefficients are a collection of weights that SIMULTANEOUSLY relate the target series to a linear combination of the specified input series. If the input series are merely observed rather than fixed via a designed experiment little can be said about the effect of any one input series (selected) on the target unless you assert and "prove" that there is no evidence of cross-correlation between the other input series and the selected input.

This is why when you add or remove an input series the coefficients will settle on another set of coefficients as @Demetri Panonos commented.

$\endgroup$
2
$\begingroup$

In some fields, it is common to discuss "shared" and "unique" covariance between an outcome (dependent) variable and the group of independent variables on which it is regressed.

If you some dependent variable y being regressed on independent variables x1, x2 and x3 it is the usual case that some of the ability of each x-variable to predict y overlaps with the predictive ability of the other x-variables. In fact, sometimes that tendency is so pronounced with say that x1, x2, and x3 are "collinear" to the extent that a multiple regression model can't easily estimate the precise effects of each x-variable on y.

An exception to this is a designed study which collects the data in such a way as to guarantee that x1, x2 and x3 are "orthogonal" which simply means the predictor variables are not correlated at all with each other. In most fields, you just don't see that sort of dataset.

So when x1, x2 and x3 are to some extent "sharing" their predictive ability for y then taking one of those x-variables out of the model causes the parameter estimates for the others to be readjusted. If both the parameter estimates and the standard errors of the parameter estimates of the remaining independent variables change a great deal when one independent variable is removed, that's a sign of the "collinearity" effect that I mentioned, above.

There are regression diagnostics such as "variance inflation factor" or "tolerance" which can be computed if you think there may be a collinearity problem. But if the estimates for the remaining parameters are only changing slightly with the removal of one variable (as usually happens) then that's just the way multiple regression models tend to work.

One bonus "food for thought" is this. If you compute the R-squared statistics for each of your independent variable predicting your dependent variable in individual simple linear regression models, then compute R-squared for the model with all the x-variables predicting y in a multiple regression model, that combined R-squared is almost always less than the sum of the individual R-squared. And in most cases it will be a lot smaller.

P.S. The answers before mine were correct and succinct. I just elaborated on the topic in case you wanted a bit more background.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.