I have one question with respect to need to use feature selection methods (Random forests feature importance value or Univariate feature selection methods etc) before running a statistical learning algorithm.
We know to avoid overfitting we can introduce regularization penalty on the weight vectors.
So if I want to do linear regression, then I could introduce the L2 or L1 or even Elastic net regularization parameters. To get sparse solutions, L1 penalty helps in feature selection.
Then is it still required to do feature selection before Running L1 regularizationn regression such as Lasso?. Technically Lasso is helping me reduce the features by L1 penalty then why is the feature selection needed before running the algo?
I read a research article saying that doing Anova then SVM gives better performance than using SVM alone. Now question is: SVM inherently does regularization using L2 norm. In order to maximise the margin, it is minimising the weight vector norm. So it is doing regularization in it's objective function. Then technically algorithms such as SVM should not be bothered about feature selection methods?. But the report still says doing Univariate Feature selection before normal SVM is more powerful.
Anyone with thoughts?