Let's assume we have infinite computing power. When we consider two algorithms,
- learning algorithm + regularization and
- feature selection + (learning algorithm + regularization),
Which one would achieve better prediction performance usually?
Now, my original post on feature selection vs regularization is duplicate with the post. I'd like to update my question focusing on the point which is still ambiguous.
I read the previous answers and they are not arriving at one conclusion. The answers point to the opinion that it depends on the situation. I'd like to narrow my question to focus on the case which still is not considered.
I summarize some of the views from the answers:
This answer mentions that it depends on the learning algorithm. It says that random forest would do better than the NN for selecting relevant features among large numbers of features. So the random forest would need feature selection less than NN.
This answer indicates that the data size for training can be the issue. The feature selection might put one more layer of training into the fitting procedure and leads to overfitting because the feature selection is done at the subset of data.
This answer mentions that if the application requires refit repeatedly with new data, then, the wrong feature might affect the performance, and feature selection would be helpful.
This answer indicates that it depends on the regularization method.
The answers consider various factors that can affect the feature selection performance. But I think there might be several still more factors to consider. In this updated question, I'd like to raise the issue on the signal to noise ratio.
My first thought is this: If the signal to noise ratio of the data set is low, there is more danger of overfitting and I feel that extra step of feature selection might help because it can remove the irrelevant feature catching the noise in a different way. In this setting, removing features might be more important than keeping more features.
However, from the opposite point of view, the extra step of feature selection might lead to more severe overfitting with argument 2.
Both views make sense to me at the moment.
How much is the feature selection(preprocessing) helpful for the learning algorithm with regularization training with the data samples of different levels of signal to noise ratios?