I have been thinking of this problem for days and I can't seem to arrive at a conclusion for feature selection in Linear Regression.
Please tell me what is wrong with this simple approach versus using more sophisticated ones like Lasso, stability, or Recursive Feature Elimination:
Include all features in StatsModel OLS --> Remove all features whose p-values are greater than 0.05 (arbitrary alpha level) --> Remaining ones are my features and I'm done.
Why use fancy algorithms like Lasso, RandomizedLasso for stability checking, and RFE at all? What am I missing?