Im working on stock price direction prediction project and have tried out some models (SVMs and Random Forests).
I used Ransom Forests for feature selection and it actually decreased the performance of Random Forests and SVMs. There was a decrease in performance in cross validation and testing both.
Does this mean that feature selection won't work at all or a different method of feature selection should be implemented?
Please help me understand the intuition behind this and any fix. Thank you.
This isn't surprising. A model with more features has a richer space of functions to approximate as compared to a function with fewer features. Feature selection, in my own opinion, is not about increasing performance. On the contrary, it is about finding a set of features which does good enough as compared to the model with the full set.
I wrote a small blog post to demonstrate that if you want increased performance, you should engineer features rather than select them. The long and the short of it is that most feature selection approaches did worse than just using the entire data set. The results are just for linear regression, but I think it tells a compelling story.
