3
$\begingroup$

Lets say i don't want to spend much time on feature engineering or thinking about what features to include/exclude, instead i include every feature that i find remotely relevant and then let lasso do the selection for me using cross validation.

I am not saying i will end up including millions of features and then make it computationally prohibitive. I am asking about the case of keeping the number of included features reasonable and then using that approach assuming that i only care about accuracy and interpretation is not of much importance to me.

Any idea/comments are welcome.

$\endgroup$
3
$\begingroup$

You care about prediction performance and not statistical inference, this is a good setting for the the LASSO and other regularisation methods (statistical interpretability of high dimensional LASSO models is tough).

You do have to be very careful about overfitting though. If you have LOTS of potential features, you'll want LOTS of training data and a sound method of selecting the best shrinkage/tuning hyperparamater. The more features you have, the more opportunities there are that at least some of them will correlate well with your response just by chance and end up being selected by the LASSO to be in the model - this will result in poor performance on out of sample data.

Consider cross-validation or setting up your own training, validation and test sets to select and validate the tuning parameter.

On a side note, modern implementations of the LASSO are actually surprisingly computationally efficient and having $10^6$ features would not be too unreasonable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.