2
$\begingroup$

I have a query regarding multiple linear regression. Let us assume that I have 3 predictor variables with a linear relationship between them as follows:

c = a + b

On inspection, I find that a,b,c are significantly correlated with each other.

  • Is it appropriate to construct a multiple linear regression with a,b,c in it?
  • Or is it more appropriate to construct two linear regressions, one with c in it and one with a,b only just to test which explain a greater proportion of variance, the whole or the sum of the parts?
  • If it is not appropriate, what is the best solution to this kind of a problem?
$\endgroup$
3
  • 2
    $\begingroup$ In short, your primary worry is about inflated variance estimates. See stats.stackexchange.com/questions/51386/… $\endgroup$ Commented Apr 2, 2013 at 20:13
  • $\begingroup$ One practical suggestion might be to try elastic net regression; if there are two explanatory variables doing the same thing, it will simply set one to zero. cran.r-project.org/web/packages/glmnet/index.html $\endgroup$ Commented Apr 2, 2013 at 20:21
  • $\begingroup$ Are you trying to compare a model $y \sim a + b$ to a model $y \sim c$? (This differs from the interpretations made by all existing answers but seems to be what the second and third bullets are asking.) $\endgroup$
    – whuber
    Commented Aug 1, 2013 at 19:41

4 Answers 4

8
$\begingroup$

If you have an exact linear relationship in your independent variables (a bit more common jargon than predictor variables) like $c=a+b$, then you cannot apply the regression purposefully. In other words, it is misspecified. A statistical software will usually come up with an error message here. Intuitively, there is no unique estimator as there is no room for variation, or in technical terms, you have an non-invertible X matrix and basically try to divide by 0.

If you have a strong correlation between lets say a and c, then this is just strong multicollinearity. You can estimate the coefficients, but you need to take into account three things:

  1. the standard deviations will be very high and the t values very low. Basically two variables try to co-explain the one dependent thing.
  2. the estimated coefficients will be very sensitive to outliers. So, data contamination is a big issue here as the coefficient can dramatically change with just one very small outlier.
  3. Given that the data is not contaminated, you need to adjust your interpretation of the coefficient. In a multiple linear OLS regression, the coefficient indicates what happens to the dependent variable if all other variables are held constant. Take for example: $$ \text{wage} = \text{constant} + \beta_1*\text{education} + \beta_2*\text{IQ} + u. $$ (Here IQ and education are expected to be strongly correlated.)

So you may suddenly wonder that your coefficient $\beta_1$ turns out to be negative, while you expect it to be theoretically by all means positive. This may be due to the very strong correlation between wage and intelligence, as intelligence may have a greater effect on wages and $\beta_1$ is downward adjusted by this effect of intelligence on wage. So, this is correct estimated, but the interpretation is now different.

However, if you take the change in the interpretation of the coefficients into account and your data correct, strong multicollinearity gives an unbiased predictor.

$\endgroup$
0
2
$\begingroup$

Statistical significance of the correlation between two predictors doesn't really tell you whether you should remove one. Indeed, with large sample size even very small correlations may be significant.

It's the effect-size (how correlated are they?) that is more relevant than the statistical significance.

If it's actually the case that $a + b = c$ (without error) then only two of them can be used, but significant correlation between say $a$ and $b$ doesn't necessarily mean both aren't useful.

$\endgroup$
2
$\begingroup$

I take it that $c$ is not just correlated with $a+b$ but is defined as $c = a+b$ (which makes sense only if $a$ and $b$ are on the same scale). Then your question is most easily answered by creating another predictor, $d = a-b$, and using $c$ and $d$ instead of $a$ and $b$ as predictors. If the coefficient of $d$ is significant then $a$ and $b$ should not receive equal weight; the change in $R^2$ associated with $d$ tells you how much it helps to weight $a$ and $b$ unequally.

$\endgroup$
0
$\begingroup$

Assuming 2 things:

  1. They are not perfectly correlated (<100%).
  2. You are not interested in interpretations, but only in good predictions

A practical (probably not too sound on a theoretical level?) way to deal with it could be to estimate the regression of c1*A+c2*B=C and keep the error terms of this regression. Then, you could use A, B and the error terms of predicting C. (which are not correlated with A and B)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.