1
$\begingroup$

I have two variables $x_1$ and $x_2$ in linear regression. I would like to see if the distance between $x_1-x_2$ is significant. So I want to add one more variable $x_3$, which is equal to $x_1-x_2$.

But does this generate dependency and thus collinearity? If so, how can I know the effect of the distance?

$\endgroup$
1
$\begingroup$

Certainly there's complete linear dependence between the existing variables $(x_1,x_2)$ and their difference ($x_3=x_1-x_2$).

One way to see this is that you can write any one of the predictors as a linear combination of the other two. Specifically, $x_3$ is defined that way, but you can also write each of the others that way. [See the third paragraph of the Wikipedia article on multicollinearity]

Note in particular, that if you add some constant $\delta$ to the coefficient of $x_3$ and also add the same constant to the coefficient of $x_2$ and subtract it from the coefficient of $x_1$, the fit will be unchanged.

As a result, you cannot uniquely identify the parameters in the larger model - any optimal solution is accompanied by an infinity of other solutions.

Without some form of constraint/regularization, you can't estimate all three at once.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.