In a prediction experiment with regularized regression methods (Ridge, Lasso, and Elastic Net), I have tried two feature-selection methods prior to running regression, and I have obtained very similar results although the feature sets were largely different as determined by the feature-selection methods.
First, I used
SelectKBest method in
scikit-learn with f_regression. With this method, I obtain a more diverse feature set, which contains some features that are inter-correlated to some degree.
Let me explain what I mean by diverse here: I can categorize the features into 5 groups based on the way they were generated (e.g., temporal behavior regarding site visits vs sequence of websites visited). With
SelectKBest method the selected features come from all different categories.
Second, I used
CfsSubsetEval method available in Weka. This method chooses features that are highly correlated with target variable while ensuring a low inter-correlation among the features selected. When I use this method (I obtain very similar results), the final set of features come majorly from one categories (sequence of websites visited), leading to limited diversity. Therefore, features regarding other activities of users (time they visit a site, count of their visits, etc.) were discarded.
At this point, I do not know which approach to follow. Does it depend on my objectives or is there a general principle to follow to make decision? For example, can I just say I prefer to use
because a good model that is based on a diverse feature set might be more generalizable to other user groups,
because the model might be more interpretable since it reflects many aspects of user behavior.
I wonder what would be a good way to approach this issue of feature selection.