In a classification problem using Linear SVM, I am trying to remove variables which have a strong correlation (Pearson) between them from a dataset.

  • What is the usual threshold recommended? I currently delete variables when they have a correlation >= 1.0 or <= -1.0 but I wonder if I should use 0.5 instead.
  • Should I create my correlation matrix after or before scaling the data ?
  • 3
    $\begingroup$ Excuse me but how do you have feature that their correlation is greater than 1? Deleting perfectly correlated features is quite reasonable but how do you get the $>$ of your heuristic? Do you somehow calculate variances and then you normalize them? In any case regrading your second question the correlation is not relevant to the scaling of your data so that should not make any difference. $\endgroup$
    – usεr11852
    Jan 4, 2016 at 17:41
  • 2
    $\begingroup$ How are you determining which variable to remove in case of high correlation? $\endgroup$ Jan 4, 2016 at 17:46
  • 1
    $\begingroup$ Keep in mind that the linear correlation coefficient only detects linear correlations. Depending on what problem you're trying to solve (unstated in the post), this additional pre-processing may not even do anything helpful. $\endgroup$
    – Sycorax
    Jan 4, 2016 at 18:17
  • $\begingroup$ @usεr11852 I don't have correlation greater than 1. I just used >= and <= so that I can change the number by another value and my condition still works in the code. Sorry if this was misleading. $\endgroup$
    – Octoplus
    Jan 4, 2016 at 18:56
  • 1
    $\begingroup$ @user777 after some test, it looks like it does not improve my classification. However it removes a lot of variables so the model is slightly faster to train. $\endgroup$
    – Octoplus
    Jan 4, 2016 at 18:58

2 Answers 2


I don't think this can be answered in the abstract, we would need to know what is the goal, and what is the variables. Some possibilities:

Two variables which basically are different measurements of the same thing with independent errors. Then keep both (or use their mean) looks reasonable. Other cases are less clear-cut. Since you have a classification problem, the following is relevant: Feature Selection: Correlation and Redundancy even highly correlated variables could have non-redundant information, and in the example given removing any of them would destroy its information content. But, interestingly, in that example replacing the two variables with their difference would work! That underscores that each example have to be treated on its own. Some general threshold value which would always work would be difficult to find!


There is no one-size-fits-all solution for this. The threshold could be judged by the researcher based on the association between the variables.

For the high correlation issue, you could basically test the collinearity of the variables to decide whether to keep or drop variables (features). You could check Farrar-Glauber test (F-G test) for multicollinearity. Here is a link to do it in R for example: https://www.r-bloggers.com/multicollinearity-in-r/

For the scaling issue, the scaling (if you mean multiplication of variables by a real number), it does not affect the linear relationship across variables. It is however recommended to scale the features for SVM if you are using Euclidean distance metric as kernel.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.