How to 'normalize' correlated factors for LASSO?

Question

When factors are correlated (in some sense, swappable), LASSO tends to select factors who predict large values (so the $\beta$ is smaller), which are not necessarily the factor are are most predictive (big values are not necessarily good values).

This is obviously not optimal, especially when we have some a-priori knowledge about the factors.

What is often suggested is to throw away those factors which are known to be correlated with other factors. However, it is hard to figure out which one to throw away. This is also something I don't understand: if I know which factor to remove, then what's the point of feature selection?

I am wondering if there is a way to keep the feature selection more controllable. The solution I can think of is to 'normalize' the factors into Z-score, which normally distributed within [-3,3] range.

Still, I am afraid this might not be robust enough. Can anyone here give me some insights?

Answer 1

Answer based on comments:

glmnet standardizes the features by z-scoring them if standardize = TRUE, which is the default. This is common practice for many implementations of the lasso.

--Andrew M

This default behavior of many implementations of LASSO is one of the approaches noted by the OP. It directly addresses the main problem here, so that the predictor-selection part of LASSO is independent of measurement scales. Normalization of predictors before LASSO means it doesn't matter whether distances, say, are measured in millimeters or miles.

glmnet, at least, then proceeds to re-scale the coefficients to match the original measurement scales. This is done silently so that it's easy to miss what's going on in the implementation.

I always use LASSO and ridge together. Adding a ridge penalty helps with these problems.

--Matthew Drury

This suggestion provides an additional aid, particularly with large numbers of potential predictors.

Note that the suggestion in another answer to pre-select a set of predictors by LASSO and then do best-subset selection among them is not advisable. That suggestion contains all the problems of automated model selection without the protection against overfitting provided by the penalization of coefficients in LASSO.

Answer 2

I don't have an answer to the first question that you ask, but for the second question, I read a paper which proposed a nice solution.

The idea is to use LASSO regression for feature filtering in the first place, then for the selected features, the best subset selection can be performed.

Details:

$\lambda$ is set to be $\lambda_{max}$ which is the smallest $\lambda$ that makes all regression coefficients zero. Then, by increasing $\lambda$, more and more regression coefficients become non-zero. Say we choose the first 40 features that are activated in such a process, then we can afford to perform the best subset selection for this smaller feature space.