9
$\begingroup$

I was reading this article related to elastic net. They say that they use elastic net because if we just use Lasso it tends to select only one predictor among the predictors that are highly correlated. But isn't this what we want. I mean it saves us from the trouble of multicollinearity doesn't it.

Any suggestions/ clarifications?

$\endgroup$
10
$\begingroup$

Suppose two predictors have a strong effect on the response but are highly correlated in the sample from which you build your model. If you drop one from the model it won't predict well for samples from similar populations in which the predictors aren't highly correlated.

If you want to improve the precision of your coefficient estimates in the presence of multicollinearity you have to introduce a little bias, off-setting it by a larger reduction in variance. One way is by removing predictors entirely—with LASSO, or, in the old days, stepwise methods—, which is setting their coefficient estimates to zero. Another is by biasing all of the estimates a bit—with ridge regression, or, in the old days, regressing on the first few principal components. A drawback of the former is that it's very unsafe if the model will be used to predict responses for predictor patterns away from those that occurred in the original sample, as predictors tend to get excluded just because they're not much use together with other, nearly collinear, predictors. (Not that extrapolation is ever completely safe.) The elastic net is a mixture of the two, as @user12436 explains, & tends to keep groups of correlated predictors in the model.

$\endgroup$
  • $\begingroup$ Why won't it predict well in this new sample? $\endgroup$ – user31820 Aug 27 '13 at 16:26
  • 1
    $\begingroup$ Because the model's missing an important predictor. $\endgroup$ – Scortchi Aug 27 '13 at 17:56
  • 2
    $\begingroup$ If two predictors are correlated in one representative sample from a population, should they not be correlated in another sample? if you use a model on data that is "away from those that occurred in the original sample", isn't that a borderline invalid use of any model? $\endgroup$ – Matthew Drury Jun 2 '16 at 5:28
  • $\begingroup$ @MatthewDrury: Well if the model's "right" - if there are no unobserved confounders worth bothering about, & if the functional form is extrapolatable - then the distribution of predictors in the sample doesn't matter (though of course it determines the precision of estimates & predictions). So at one extreme you might have a mechanistic model built on data from a well-controlled experimental study on causal factors; at the other an empirical model built on data collected from an observational study on a bunch of variables that were merely easy to measure. $\endgroup$ – Scortchi Jun 2 '16 at 13:57
  • $\begingroup$ The phrase: "in the old days, stepwise methods made me smile. :D (Obvious +1, this is a good answer) $\endgroup$ – usεr11852 Nov 13 '16 at 19:17
4
$\begingroup$

But isn't this what we want. I mean it saves us from the trouble of multicollinearity isn't it.

Yes! and no. Elastic net is a combination of two regularization techniques, the L2 regularization (used in ridge regression) and L1 regularization (used in LASSO).

Lasso produces naturally sparse models, i.e. most of the variable coefficients will be shrinked to 0 and effectively excluded out of the model. So the least significant variables are shrinked away, before shrinking the others, unlike with ridge, where all variables are shrinked, while none of them are really shrinked to 0.

Elastic net uses a linear combination of both these approaches. The specific case mentioned by Hastie when discussing the method was in the case of large p, small n. Which means: high dimensional data with, relatively few observations. In this case LASSO would (reportedly) only ever select at most n variables, while eliminating all the rest, see paper by Hastie.

It will always depend on the actual dataset, but you can well imagine that you don't always want to have the upper limit on the number of variables in your models being equal to, or lower than the number of your observations.

$\endgroup$
  • $\begingroup$ But what about multicollinearity. Elastic net does allow to select multi collinear features which is not good isn't it? $\endgroup$ – user31820 Aug 27 '13 at 16:28
  • $\begingroup$ I don't think that many real datasets have perfectly multicollinear variables. Highly correlated variables might be nearly collinear, which is still a problem, but one that you might be willing to accept, in case they are both important for your model. $\endgroup$ – means-to-meaning Aug 27 '13 at 21:49
  • $\begingroup$ The link added above leads to yahoo.com. Also, [the paper] (onlinelibrary.wiley.com/doi/10.1111/j.1467-9868.2005.00503.x/…) is by Zou and Hastie (Elastic net one). $\endgroup$ – KarthikS Oct 25 '16 at 18:09
2
$\begingroup$

Both Lasso and Elastic Net are efficient methods to perform variable or feature selection in high-dimensional data settings (much more variables than patients or samples; e.g., 20,000 genes and 500 tumor samples).

It has been shown (by Hastie and others) that Elastic Net can outperform Lasso when the data is highly correlated. Lasso may just select one of the correlated variables and does not care which one is selected. This can be a problem when one wants to validate the selected variables in an independent dataset. The variable selected by Lasso may not be the best predictor among all correlated variables. Elastic Net solves this problem by averaging highly correlated variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.