4
$\begingroup$

Do you know of any techniques that allows one to avoid and get rid of multicolinearity in SVM input data? We all know that if multicolinearity exists, explanatory variables have a high degree of correlation between themselves which is problematic in all regression models (the data matrix is not invertible and so on).

My question is actually a bit more subtle as relevant data has to be selected. In other words, multicolinearity must be avoided while keeping relevant input data. So in a way, I guess that my question is also about reducing the input data matrix dimension.

So, I've thought about PCA, which permits both reducing dimension and getting uncorrelated vectors (the PCs), but then I don't know how to deal with the eigenvector's signs.

Improve this question
$\endgroup$
8
  • $\begingroup$ If multi-colinearity is high then when you plot the sorted eigenvalues of the SVD (of the standardized covariance matrix) you should see a number of small values near the end. You can zero these out, and form a basis with reduced dimension. I don't know why the eigenvalue's sign is a problem. $\endgroup$
    – muratoa
    Commented Sep 5, 2012 at 9:16
  • $\begingroup$ Also I don't think SVMs are too sensitive to multicollinearity, especially when used with kernels. If some dimensions are collinear, they will contribute identically to the pairwise kernel distances but the SVM will just seperate on the dimensions that do contribute. Multi-colinearity is more of an issue when $(X^tX)^{-1}$ is necessary for the fit/inference. $\endgroup$
    – muratoa
    Commented Sep 5, 2012 at 9:20
  • $\begingroup$ Ok, so If I consider a given dataset, I carry out a PCA on its covariance matrix, then extract de most significant PCs and use them as input data for my SVM. Am I right? $\endgroup$
    – marino89
    Commented Sep 5, 2012 at 9:20
  • 1
    $\begingroup$ Yeah you can. But try using the full data and then try reducing with different (sum total) eigenvalue thresholds and see if there is a significant difference. Because if you fit an SVM with a transformed basis you have to apply this transform on each new data point when used for prediction. $\endgroup$
    – muratoa
    Commented Sep 5, 2012 at 9:22
  • 2
    $\begingroup$ Just a note that multicollinearity can exist even if there is no pair of variables that are highly correlated. Multicollinearity is a trait of sets of data. E.g., if you have 11 variables and one is a sum of the other 10, you will have perfect collinearity, but no high correlations: $\endgroup$
    – Peter Flom
    Commented Sep 5, 2012 at 10:53

1 Answer 1

Reset to default
9
$\begingroup$

Multicolinearity is not generally a problem for SVMs. Ridge regression is often used where multicolinearity is an issue, as the regularisation term resolves the invertibility issue by adding a ridge. The SVM uses the same regularisation term as ridge regression does, but with the hinge loss in place of the squared error.

Ridge regression has a link with PCA as explained by Tamino, essentially it penalises principal components with large eigenvalues less than components with low eigenvalues, so it is a bit like having a soft selection of the most important PCs, rather than a hard (binary) selection.

The important thing is to make sure that the regularisation parameter, C, and any kernel parameters are tuned correctly and carefully, preferably by minimising an appropriate cross-validation based model selection criterion. This really is the key to getting good results with an SVM (and kernel methods in general).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.