Linear Regression Feature selection: multiple-regression p-value filter versus Lasso / Recursive Feature Elimination

Question

I have been thinking of this problem for days and I can't seem to arrive at a conclusion for feature selection in Linear Regression.

Please tell me what is wrong with this simple approach versus using more sophisticated ones like Lasso, stability, or Recursive Feature Elimination:

Include all features in StatsModel OLS --> Remove all features whose p-values are greater than 0.05 (arbitrary alpha level) --> Remaining ones are my features and I'm done.

Why use fancy algorithms like Lasso, RandomizedLasso for stability checking, and RFE at all? What am I missing?

Answer 1

Feature significance does not equal to an increased cross-validation score. I've been researching this quite extensively lately. I've had many significant features reduce my cross-validation score, and have had some non significant features increase CV score on K-folds cross validation. It all depends on what you want to achieve, if its a valid CV score, then you'll need to proceed with other methods.