I was reading this article related to elastic net. They say that they use elastic net because if we just use Lasso it tends to select only one predictor among the predictors that are highly correlated. But isn't this what we want. I mean it saves us from the trouble of multicollinearity doesn't it.
Any suggestions/ clarifications?
Suppose two predictors have a strong effect on the response but are highly correlated in the sample from which you build your model. If you drop one from the model it won't predict well for samples from similar populations in which the predictors aren't highly correlated.
If you want to improve the precision of your coefficient estimates in the presence of multicollinearity you have to introduce a little bias, off-setting it by a larger reduction in variance. One way is by removing predictors entirely—with LASSO, or, in the old days, stepwise methods—, which is setting their coefficient estimates to zero. Another is by biasing all of the estimates a bit—with ridge regression, or, in the old days, regressing on the first few principal components. A drawback of the former is that it's very unsafe if the model will be used to predict responses for predictor patterns away from those that occurred in the original sample, as predictors tend to get excluded just because they're not much use together with other, nearly collinear, predictors. (Not that extrapolation is ever completely safe.) The elastic net is a mixture of the two, as @user12436 explains, & tends to keep groups of correlated predictors in the model.
But isn't this what we want. I mean it saves us from the trouble of multicollinearity isn't it.
Yes! and no. Elastic net is a combination of two regularization techniques, the L2 regularization (used in ridge regression) and L1 regularization (used in LASSO).
Lasso produces naturally sparse models, i.e. most of the variable coefficients will be shrinked to 0 and effectively excluded out of the model. So the least significant variables are shrinked away, before shrinking the others, unlike with ridge, where all variables are shrinked, while none of them are really shrinked to 0.
Elastic net uses a linear combination of both these approaches. The specific case mentioned by Hastie when discussing the method was in the case of large p, small n. Which means: high dimensional data with, relatively few observations. In this case LASSO would (reportedly) only ever select at most n variables, while eliminating all the rest, see paper by Hastie.
It will always depend on the actual dataset, but you can well imagine that you don't always want to have the upper limit on the number of variables in your models being equal to, or lower than the number of your observations.
Both Lasso and Elastic Net are efficient methods to perform variable or feature selection in high-dimensional data settings (much more variables than patients or samples; e.g., 20,000 genes and 500 tumor samples).
It has been shown (by Hastie and others) that Elastic Net can outperform Lasso when the data is highly correlated. Lasso may just select one of the correlated variables and does not care which one is selected. This can be a problem when one wants to validate the selected variables in an independent dataset. The variable selected by Lasso may not be the best predictor among all correlated variables. Elastic Net solves this problem by averaging highly correlated variables.
