Just an early warning before you read the rest, I am probably missing some theory behind it and I have lack of knowledge that needs to be filled with this question. So, what I will say might be very basic and may sound stupid!

As far as I see, in machine learning tasks, a common approach is to generate a bunch of features that could possible tell something about the outcome variable (y) and therefore contribute to the prediction power of a regression model, without checking if they indeed significantly correlate with y.

I wonder why one would not use correlation to filter some features before including anything in a regression model. In simple logic, if two things are not significantly associated, then why on earth one should predict the other?

Or, including a bunch of features is common since many people for example count on some regularization methods that sort of performs an internal feature selection?

  • $\begingroup$ Problem is that a feature $x$ might have zero correlation with the response,. But still be i portant in cmbination with other variables $\endgroup$ – kjetil b halvorsen Feb 25 '17 at 19:27
  • $\begingroup$ Adding to @kjetil, one way that can happen is if one variable "suppresses" the irrelevant portion of another variable. Here are some examples: home.ubalt.edu/tmitch/645/articles/… $\endgroup$ – David Lane Feb 26 '17 at 0:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.