I am new to machine learning.
I have a basic question about feature selection. I have a dataset with 100 features which I used to regress an output Variable. When I do regression with all the features, I get a particular regression error, r1. When I do feature selection (using step forward feature selection) and select X (which is less than 100) features, I get a better regression r2 (r2 << r1).
I am trying to answer the question how the machine learning algorithm is doing worse with more features. Isn't the performance of the algorithm supposed to increase (or at least remain the same) when we add new features? Does it mean that the algorithm is not a good choice for the problem or does it mean that I don't have enough data for the algorithm to learn?
Can you please help me?