I have been thinking of this problem for days and I can't seem to arrive at a conclusion for feature selection in Linear Regression.

Please tell me what is wrong with this simple approach versus using more sophisticated ones like Lasso, stability, or Recursive Feature Elimination:

Include all features in StatsModel OLS --> Remove all features whose p-values are greater than 0.05 (arbitrary alpha level) --> Remaining ones are my features and I'm done.

Why use fancy algorithms like Lasso, RandomizedLasso for stability checking, and RFE at all? What am I missing?


Feature significance does not equal to an increased cross-validation score. I've been researching this quite extensively lately. I've had many significant features reduce my cross-validation score, and have had some non significant features increase CV score on K-folds cross validation. It all depends on what you want to achieve, if its a valid CV score, then you'll need to proceed with other methods.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.