I'm working with a group on an ML project and one of the team members has proposed using L1 to reduce the feature space and then apply an exhaustive search with the reduced feature set.

In each step, k-fold cross-validation is applied with the same hold-out sets for each k.

Is this approach sound?

  • 1
    $\begingroup$ What is meant by an exhaustive search? Do you mean a two step feature selection procedure? you haven’t specified the model in the second stage $\endgroup$ – Tim Mak Aug 22 '20 at 8:22
  • $\begingroup$ Both stages use logistic regression. By exhaustive search I mean to try every combination of features. $\endgroup$ – jeffalltogether Aug 22 '20 at 18:21
  • $\begingroup$ Like @Tylerr said, it really depends on the goal of your analysis. For prediction, you're generally better off with other approaches than regression. If it is not for prediction, then reducing the feature space always comes with the risk of losing some important variables. The best course of action again depends on exactly what you want to do. $\endgroup$ – Tim Mak Aug 24 '20 at 1:54

It will have the desired out of reducing your feature space, but why not just use the output from the L1 with some tuning of the reg. parameter? Does it not provide good cv accuracy? Are you just taking the parameters chosen and putting it into another model? Do you require even more dimensionality reduction? Exhaustive search will be extremely computationally expensive depending on the number of params.

Maybe look at methods which will handle tons of redundant variables naturally such as random forests or boosted trees if you need prediction accuracy.

If your goal is explanatory power, I personally do not like exhaustive methods for variable selection and recommend going the bayesian route which have a lot of nice sparse priors to handle this scenario.

  • $\begingroup$ Thanks for the feedback. My main concern is data leakage. That maybe the second feature selection is biased by the first. I don’t know the answer to your other questions at the moment, but will look into them. Thank you! $\endgroup$ – jeffalltogether Aug 22 '20 at 15:53

You are likely to face problems with your exhaustive approach. Suppose we just generated random noise for all your covariates. Let's say we have 10 covariates. So that yields $2^{10}=1024$ possible models using just those covariates (so no interactions or products or ratios).

If you test your models at a 5% level, you should expect about 51 of those noise models to appear significant -- and yet you will have found nothing truly significant. Out-of-sample, we would expect those models to perform poorly. With your data, you should not necessarily expect much better performance than the noise models out-of-sample.

Ah, but you are working with $k$-fold cross validation! Great... except that is no guarantee of better performance either. Maybe you will find fewer false-positive models. However, you are not much more protected because the data that is held out for "out-of-sample" testing will guide your choices for a penalty and thus a model. Thus that out-of-sample data effectively becomes in-sample. You can again expect poor performance for your model on true out-of-sample data.

Yes, some people will claim you need to use ML techniques and not regression for prediction. Nonsense. Look at many comparisons of modeling where they are then measured on true out-of-sample data and you will see that simple regressions often outperform more complicated methods.

The sad truth is that feature selection is hard, as in NP-hard but also as in intellectually difficult regardless of computing power. That is the nature of randomness -- so you should not expect that the proposed LASSO-like approach will be much better than other modeling approaches.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.