1
$\begingroup$

I am performing feature selection using 3 different techniques Lasso, Random forest and gradient boosting. I am comparing these 3 results based on Test MSE (Using CV). I pick the one that has the lowest MSE and get the important variables. The dataset I am using is standardized before performing feature selection.

Of the variables that get picked as important ones, there is one that stands out to be the most important one compared to the rest. My question is, is it valid to perform subset selection after feature selection to find a model that has fewer parameters than the features that get selected by the algorithms above? If not, is there another technique to find a narrow set of selected features?

$\endgroup$
1
$\begingroup$

Yes. Keep in mind, machine learning algorithms are generalized to be data driven and rely heavily on sample data, and often do not incorporate domain knowledge. Many feature selection techniques are great, but you still need to use your head (domain knowledge). A common technique is to use Boruta All-Relevant feature selection to identify all of the important coavariates, then you could apply Recursive Feature Elimination to find a minimal-optimal covariate set. You could go one step further and build your own set using domain knowledge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.