Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In a classification problem using Linear SVM, I am trying to remove variables which have a strong correlation (Pearson) between them from a dataset.
I don't think this can be answered in the abstract, we would need to know what is the goal, and what is the variables. Some possibilities:
Two variables which basically are different measurements of the same thing with independent errors. Then keep both (or use their mean) looks reasonable. Other cases are less clear-cut. Since you have a classification problem, the following is relevant: Feature Selection: Correlation and Redundancy even highly correlated variables could have non-redundant information, and in the example given removing any of them would destroy its information content. But, interestingly, in that example replacing the two variables with their difference would work! That underscores that each example have to be treated on its own. Some general threshold value which would always work would be difficult to find!
There is no one-size-fits-all solution for this. The threshold could be judged by the researcher based on the association between the variables.
For the high correlation issue, you could basically test the collinearity of the variables to decide whether to keep or drop variables (features). You could check Farrar-Glauber test (F-G test) for multicollinearity. Here is a link to do it in R for example: https://www.r-bloggers.com/multicollinearity-in-r/
For the scaling issue, the scaling (if you mean multiplication of variables by a real number), it does not affect the linear relationship across variables. It is however recommended to scale the features for SVM if you are using Euclidean distance metric as kernel.
