4
$\begingroup$

In a classification problem using Linear SVM, I am trying to remove variables which have a strong correlation (Pearson) between them from a dataset.

  • What is the usual threshold recommended? I currently delete variables when they have a correlation >= 1.0 or <= -1.0 but I wonder if I should use 0.5 instead.
  • Should I create my correlation matrix after or before scaling the data ?
$\endgroup$
  • 3
    $\begingroup$ Excuse me but how do you have feature that their correlation is greater than 1? Deleting perfectly correlated features is quite reasonable but how do you get the $>$ of your heuristic? Do you somehow calculate variances and then you normalize them? In any case regrading your second question the correlation is not relevant to the scaling of your data so that should not make any difference. $\endgroup$ – usεr11852 Jan 4 '16 at 17:41
  • 2
    $\begingroup$ How are you determining which variable to remove in case of high correlation? $\endgroup$ – Jason Morgan Jan 4 '16 at 17:46
  • 1
    $\begingroup$ Keep in mind that the linear correlation coefficient only detects linear correlations. Depending on what problem you're trying to solve (unstated in the post), this additional pre-processing may not even do anything helpful. $\endgroup$ – Sycorax Jan 4 '16 at 18:17
  • $\begingroup$ @usεr11852 I don't have correlation greater than 1. I just used >= and <= so that I can change the number by another value and my condition still works in the code. Sorry if this was misleading. $\endgroup$ – Octoplus Jan 4 '16 at 18:56
  • 1
    $\begingroup$ @user777 after some test, it looks like it does not improve my classification. However it removes a lot of variables so the model is slightly faster to train. $\endgroup$ – Octoplus Jan 4 '16 at 18:58
1
$\begingroup$

I don't think this can be answered in the abstract, we would need to know what is the goal, and what is the variables. Some possibilities:

Two variables which basically are different measurements of the same thing with independent errors. Then keep both (or use their mean) looks reasonable. Other cases are less clear-cut. Since you have a classification problem, the following is relevant: Feature Selection: Correlation and Redundancy even highly correlated variables could have non-redundant information, and in the example given removing any of them would destroy its information content. But, interestingly, in that example replacing the two variables with their difference would work! That underscores that each example have to be treated on its own. Some general threshold value which would always work would be difficult to find!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.