Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm working with a group on an ML project and one of the team members has proposed using L1 to reduce the feature space and then apply an exhaustive search with the reduced feature set.
In each step, k-fold cross-validation is applied with the same hold-out sets for each k.
Is this approach sound?
It will have the desired out of reducing your feature space, but why not just use the output from the L1 with some tuning of the reg. parameter? Does it not provide good cv accuracy? Are you just taking the parameters chosen and putting it into another model? Do you require even more dimensionality reduction? Exhaustive search will be extremely computationally expensive depending on the number of params.
Maybe look at methods which will handle tons of redundant variables naturally such as random forests or boosted trees if you need prediction accuracy.
If your goal is explanatory power, I personally do not like exhaustive methods for variable selection and recommend going the bayesian route which have a lot of nice sparse priors to handle this scenario.
You are likely to face problems with your exhaustive approach. Suppose we just generated random noise for all your covariates. Let's say we have 10 covariates. So that yields $2^{10}=1024$ possible models using just those covariates (so no interactions or products or ratios).
If you test your models at a 5% level, you should expect about 51 of those noise models to appear significant -- and yet you will have found nothing truly significant. Out-of-sample, we would expect those models to perform poorly. With your data, you should not necessarily expect much better performance than the noise models out-of-sample.
Ah, but you are working with $k$-fold cross validation! Great... except that is no guarantee of better performance either. Maybe you will find fewer false-positive models. However, you are not much more protected because the data that is held out for "out-of-sample" testing will guide your choices for a penalty and thus a model. Thus that out-of-sample data effectively becomes in-sample. You can again expect poor performance for your model on true out-of-sample data.
Yes, some people will claim you need to use ML techniques and not regression for prediction. Nonsense. Look at many comparisons of modeling where they are then measured on true out-of-sample data and you will see that simple regressions often outperform more complicated methods.
The sad truth is that feature selection is hard, as in NP-hard but also as in intellectually difficult regardless of computing power. That is the nature of randomness -- so you should not expect that the proposed LASSO-like approach will be much better than other modeling approaches.
