MUST we ALWAYS remove the variables which are linear combinations of others? I am always told to remove the variables which are linear combination of other variables. This makes sense for me for the methods which need to use covariance matrix, for example linear regresson, if we don't remove such variables, there will be problem of ranks, inversibility, etc. However for other methods which don't make use of covariance matrix (e.x. boosting, tree), does this practice still hold?
 A: If you know the variables are linear combos of other variables then you should remove them. They do not add anything, but only make everything a little harder, in some cases, such as inference and OLS estimation, quite a bit harder.
A: For a cluster analysis, you may be able to run the analyses with linearly dependent variables. However, the cluster solution will most likely attach extra (or less) weight to the variables associated the linear combination...particularly if a distance metric is being used. If this is desired, then it may be reasonable. Otherwise, I cannot recall any references that recommend such a move.
That said, I have been curious about non-horizontal/non-vertical linear breaking patterns in CART analyses. My inclination is that such an analysis would require the linear combinations of subsets of the variables being used in the clustering process. However, I have not fully explored this enough to be able to answer definitively. Hopefully others with more experience in this domain can share their thoughts.
