I am working on a high-imbalanced (80-20 ratio), high-dimensional dataset (200 sample size, 300 features) where all variables are highly correlated. Can I remove the perfectly correlated (using Pearson's correlation coefficient) variables (6 variables) before cross-validation? Does this cause any information leakage?


1 Answer 1


Removing perfectly correlated variables (pairwise Pearson coefficient) is fine. It's like removing a constant feature.

  • $\begingroup$ What, exactly, are "perfectly correlated variables"? It seems obvious until you contemplate higher-dimensional problems in which, say, $300$ features with $200$ observations might yield a rank-$100$ matrix. Which variables ought to be removed and why? Indeed, what is getting cross-validated in this context? Do the correlations include those between the "variables" and some unstated response variable or do they include only correlations among the explanatory variables alone? There's an awful lot that matters here which is left to the imagination! $\endgroup$
    – whuber
    May 8, 2023 at 15:41
  • $\begingroup$ I believe I've just assumed +/-1 Pearson correlation coefficient between two features, i.e. linear dependence. $\endgroup$
    – gunes
    May 8, 2023 at 15:44
  • $\begingroup$ Isn't the question about 300 features, not just two? Their covariance matrix can have rank as small as $2$ without any correlation coefficient being close to $\pm 1.$ $\endgroup$
    – whuber
    May 8, 2023 at 15:55
  • 3
    $\begingroup$ The standard approach is to regularize the problem (e.g., using a Lasso or Bayesian method) rather than proposing some ad hoc method of removing variables. But you appear to contradict yourself: at one point you refer to "300 features" and later you refer to "6 variables". Are "variables" and "features" the same or not? (For some people, a "variable" is a group of features.) Note, too, that inevitably there will be at least 300 - 200 = 100 features that are perfectly linearly dependent on the remaining features. $\endgroup$
    – whuber
    May 8, 2023 at 17:38
  • 1
    $\begingroup$ Perfectly valid points. At least, I've tried to make my point clear by just considering pairwise correlations (assuming it was being asked). But, a more general treatment of linear dependence between groups of features requires the covariance matrix to be accounted and is more complicated than what I've just scribbled. $\endgroup$
    – gunes
    May 8, 2023 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.