Say I have a dataset $D$ with $N$ features that are trying to predict a target $y$. I would like to build a model from $D$ and part of that process is removing correlated columns to reduce redundancy.

If $D$ remains constant, would changing the target $y$ ever change the method I use to check for correlation within the dataset $D$?


For an example of what I mean by redundancy see: https://arxiv.org/abs/1908.05376. I'm not interested in the relevancy part of the paper.


Say I'm using dataset $D$ to train a classification model. As part of preprocessing I check for correlations using method $M$, which could be any type of correlation algorithm, provided $M$ is unsupervised.

I choose one column from each correlated group at random. In other words, I select columns in an unsupervised fashion.

Should I ever change $M$ if I switch from a classification to a regression model, changing $y$ in the process?

Pre-empting XY

This is intended as a general question, which will lead to a specific question. The content of the specific question will depend on the answer to this question. Therefore, I believe it is not XY.

  • $\begingroup$ Of course. Consider two circumstances: (1) $y$ is uncorrelated with $D$ and (2) $y$ is a multiple of a single feature. This is discussed at length in threads about principal components (especially those that might reference regression), including stats.stackexchange.com/questions/9590, stats.stackexchange.com/questions/444545. But your question does sound like an XY problem because you don't explain what your "noise" might be, why it might be a problem, or why you believe the solution is to remove some columns outright. $\endgroup$
    – whuber
    May 22, 2023 at 17:53
  • 1
    $\begingroup$ @whuber It depends on what they mean by "removing correlated columns", it's unclear. If they want to remove features that are redundant (correlated with each other, but not necessarily correlated with the target), that can be done in total absence of the target and will of course not be affected by the $y$ values. Of course there will be dependence if they are selecting variables with respect to $y$, though. $\endgroup$ May 22, 2023 at 18:03
  • $\begingroup$ @Nuclear The link is profound, as I had hoped was evident by contemplating the two circumstances I mentioned. For a linear model, it comes down to the relationship between $y$ and the space orthogonal to that generated by $D.$ You also implicitly assume the model will be linear in the features -- but many ML models are not. $\endgroup$
    – whuber
    May 22, 2023 at 18:06
  • 1
    $\begingroup$ @whuber I'm confused by your assertion, one can identify and eliminate redundant features even if there is no target variable. Not only do we not care what values $y$ takes, we don't even care if $y$ exists. I agree it's often useful to select features that will be informative for the particular downstream prediction task, but the OP seems to be explicitly selecting features irrespective of how they relate to $y$. $\endgroup$ May 22, 2023 at 20:09
  • 1
    $\begingroup$ It absolutely would change, because there are many methods and what you would do would depend on what model you are thinking of and what you know about $y.$ Apart from identifying perfect linear relations among $D,$ which is a purely mathematical exercise, the kinds of near-correlations ("multicollinearity") you might look for and address depend on all that context. One difficulty I have is that you seem to be posing a counter-factual situation in which you aim to model $y$ but then tell us to ignore $y.$ At that point it seems pointless to do anything. $\endgroup$
    – whuber
    May 22, 2023 at 22:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.