Was discussing with a friend: suppose we have one model that uses 1,000 features and another that uses 100,000 features. Assuming their first 1,000 features are the same, shouldn't the one with 100,000 features always do at least as well as the 1,000 feature model?
I say this because, if there's a correlation between an additional feature and the target variable, it can learn this. If there's no correlation, the model should learn to ignore. So more features should always be at least as good as a model with only a subset of the same features.
My friend claims features can actively hamper model performance so that more isn't always better...how is this possible?
Thanks!