Having a dataset of 1000 data items, with 25 attributes (a mixture of numerical and categorical attributes), a binary label and using random forest classifier (it's a cost-sensitive learning task as data is a bit imbalanced, if that matters) I perform backward feature elimination which results in 20 attributes and a tangible performance decrease.

What are some potential reasons for this happening? Let's say I have done a somewhat good job in manual feature selection and extraction having the domain knowledge. Does this decrease mean that automatic feature selection might be inappropriate having manual feature selection done, or something might be wrong with the way I'm doing automatic feature selection? If the answer is highly dependent on data, then I'm looking for some potential reasons that might be causing this.