I'm learning about feature selection. I can see why it would be important and useful, for model-building. But let's focus on supervised learning (classification) tasks. Why is feature selection important, for classification tasks?
I see lots of literature written about feature selection and its use for supervised learning, but this puzzles me. Feature selection is about identifying which features to throw away. Intuitively, throwing away some features seems self-defeating: it is throwing away information. It seems like throwing information shouldn't help.
And even if removing some features does help, if we are throwing away some features and then feeding the rest into a supervised learning algorithm, why do we need to do that ourselves, rather than letting the supervised learning algorithm handling it? If some feature is not helpful, shouldn't any decent supervised learning algorithm implicitly discover that and learn a model that doesn't use that feature?
So intuitively I would have expected that feature selection would be a pointless exercise that never helps and can sometimes hurt. But the fact that it's so widely used and written about makes me suspect that my intuition is faulty. Can anyone provide any intuition why feature selection is useful and important, when doing supervised learning? Why does it improve the performance of machine learning? Does it depend upon which classifier I use?