# Correlation of features in binary classification

I read somewhere that, in binary classification problems, very strong correlation does not imply redundancy of features, for example, if $$X_i$$ and $$X_j$$ have a correlation coefficient $$\rho> 0.95$$, dropping $$X_j$$ might lead to losing information and thus make the classification model less accurate.

Is that true and if so, is the correlation matrix of features of any use if you cannot drop highly correlated features?

Further info about my problem: Classifying tuples of $$50$$ values as either signal or background. Many variables have correlation higher than $$0.8$$ and some have even stronger ($$> 0.9$$). I doubt that dropping those variables is the right thing to do, but I cannot explain it in theory.

• It depends on the type of approach you ae going to use in buiding your classifier. GLM may suffer from very high correlation. Decision-tree based methods, and neural neworks, do not need the removal of redundant features, instead they willgain additional information from them. Commented Jan 31, 2021 at 9:40
• If the correlated features are precisely measured they might be very useful, while if they are noisy maybe not? And, how to decide which to drop, if any? Maybe think about regularization as an alternative? Commented Jan 31, 2021 at 13:54

In general, even highly correlated features can carry independent information. Consider the case where we have a target variable that simply represents the equality of two variables $$X_1$$ and $$X_2$$. No matter how highly correlated those variables are, it is impossible to predict the target at a rate better than random by using only one of the variables.
The trouble is, it's hard to know how much the "unique" signal in each variable is actually contributing to the target value. If the target variable is determined only by $$X_1$$, you'd be justified in dropping all other variables even if uncorrelated, but as the counterexample shows, sometimes you can't even drop the highly correlated features and expect to maintain predictive performance. I don't think this is anything unique to binary classification problems, it should extend to multi-class and regression problems as well - similarity in the input feature space doesn't preclude the possibility that the "independent parts" of imperfectly but even highly correlated features can be useful in predicting the target.