2
$\begingroup$

I am always told to remove the variables which are linear combination of other variables. This makes sense for me for the methods which need to use covariance matrix, for example linear regresson, if we don't remove such variables, there will be problem of ranks, inversibility, etc. However for other methods which don't make use of covariance matrix (e.x. boosting, tree), does this practice still hold?

$\endgroup$
1
  • $\begingroup$ Even in a tree-based model having correlated variables is problematic because it makes the fitting process more unstable. If anything this makes one of the core issues of trees, which is that small changes in the data lead to big changes in the fitted structures, worse. In addition if you dealing with boosting and you try to compute variable importance the coefficients will be misleading. $\endgroup$
    – usεr11852
    Mar 26, 2018 at 20:46

2 Answers 2

1
$\begingroup$

If you know the variables are linear combos of other variables then you should remove them. They do not add anything, but only make everything a little harder, in some cases, such as inference and OLS estimation, quite a bit harder.

$\endgroup$
6
  • $\begingroup$ I understand the reason to do this for OLS estimation. However, what about boosting, tree? or any other methods which don't make use of covariance matrix, what is the reason to NOT include such variables? $\endgroup$
    – SiXUlm
    Mar 26, 2018 at 19:40
  • $\begingroup$ What does it add to your boosting tree? Give me one benefit of having them $\endgroup$
    – Aksakal
    Mar 26, 2018 at 19:41
  • 1
    $\begingroup$ I have no reason to include, but I also have no reason to exclude. That's why I'm asking about whether we should or should not do it. $\endgroup$
    – SiXUlm
    Mar 26, 2018 at 19:43
  • 1
    $\begingroup$ Personally, I agree with you because I always do it. But suddenly I wonder why I should do that. $\endgroup$
    – SiXUlm
    Mar 26, 2018 at 19:48
  • $\begingroup$ Consider $z = x_1 + x_2, \space y = 0 \text{ if z<0, } 1 \text{ otherwise}.$. A tree-based model will be much better off if $z$ is present as an option, but a priori you might well not know that, and want to include both $x_1$ and $x_2$ as well. In this simple example the tree-based model will certainly pick $z$ to branch on, even with a fair amount of randomness, at least as long as both $x_1$ and $x_2$ contribute substantially to the variability of $z$. $\endgroup$
    – jbowman
    Mar 26, 2018 at 19:50
0
$\begingroup$

For a cluster analysis, you may be able to run the analyses with linearly dependent variables. However, the cluster solution will most likely attach extra (or less) weight to the variables associated the linear combination...particularly if a distance metric is being used. If this is desired, then it may be reasonable. Otherwise, I cannot recall any references that recommend such a move.

That said, I have been curious about non-horizontal/non-vertical linear breaking patterns in CART analyses. My inclination is that such an analysis would require the linear combinations of subsets of the variables being used in the clustering process. However, I have not fully explored this enough to be able to answer definitively. Hopefully others with more experience in this domain can share their thoughts.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.