Importance of removing correlated predictors in non linear regression and classification Removing linear combinations and correlated predictors from a dataset is something I do when I have to fit a linear regression. I wonder though how this technique affects in general the outcome of e.g. SVM or Random Forest.
 A: In principal it shouldn't affect the performance of an SVM very much, because the SVM uses regularisation to avoid over-fitting (c.f. ridge regression), which is also effective in making the optimisation problem well posed once more.  However, the problem is then to find a good value for the regularisation parameter, $C$.
Also the SVM is based on an approximation of a bound on generalisation performance that is independent of the dimensionality of the feature space, so at least for linear SVMs, it would be reasonable for it to have little effect.  For non-linear SVMs, it will depend on the sensitivity of the kernel to the correlation.  For the common RBF kernel, it will mean that the induced feature space will be a little more sensitive to a particular direction in the input space, but probably not enough for it to be problematical.
In my experience, it is all too easy to make the performance of kernel methods worse by manually tinkering with the input representation in order to improve performance.  Generally this will just end up over-fitting the test data, as the operator has become part of the model selection procedure in a way that is not properly taken into account in performance estimation.  A good example of this is provided by machine learning challenges (e.g. kaggle) that have a leader board.  Quite often those that work hard to move up the leaderboard find themselves near the bottom of the final rankings (which are determined on a separate sample of data).  The reason is often "over-tuning".
A: In linear regression model, 
$\mathbf{y} = \mathbf{X}\beta$
the unique solution for $\beta$ is,
$\hat{\beta} = (\mathbf{X}^T\mathbf{X})^{-1} \mathbf{X}^T \mathbf{y}$
where we need to calculate the inverse of the obtained matrix: $\mathbf{X}^T\mathbf{X}$. Note that the resulted for a solution to exist, the matrix must be non-singular or in other words, the basis vectors must be linearly independent. Therefore, any redundant basis model must be eliminated but sometimes the data behaves in a way that the $\mathbf{X}^T\mathbf{X}$ becomes very close to being singular. In this situation, one solution is to use singular value decomposition (SVD). 
