0
$\begingroup$

If I have a set of features $\left \{x_1,x_2,x_3 \right \}$ and I expand the feature set by multiplying all the features to have the following: $\left \{x_1,x_2,x_3,x_1\times x_2,x_2\times x_3,x_1\times x_3 \right \}$. Now, I find that two of my features $\left \{x_1\times x_2, x_2\times x_3 \right\}$ are highly correlated ($>0.85$ spearman/pearson). What should I do? Note that originally $x_1, x_2$ and $x_3$ are not correlated at all ($<0.5$)

And I am training Logistic Regression/SVM with these features, so correlation matters.

Thanks

Improve this question
$\endgroup$
3
  • $\begingroup$ Please explain why correlation among the features matters at all! Your language of "training" suggests your objective is prediction rather than explanation; although correlation among predictors can affect an explanation, it has no effect on prediction. $\endgroup$
    – whuber ♦
    Commented Feb 28, 2020 at 0:46
  • $\begingroup$ Yes, my objective is prediction. I will train/validate/test doing K-fold cross validation and so on. But my problem comes from the fact that high correlated features make the models (weights) unstable which is usually not recommended $\endgroup$
    – Luis Pinto
    Commented Feb 28, 2020 at 14:49
  • 1
    $\begingroup$ Sometimes instability is a problem, but it depends on the extent to which the explanatory variables may differ in future samples. Whenever they are substantially outside the range used in modeling, you're taking a risk. "Instability," understood in this relative sense, is no problem at all, because it inheres in all such predictive modeling efforts. $\endgroup$
    – whuber ♦
    Commented Feb 28, 2020 at 14:52

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.