1
$\begingroup$

I'm getting started with regularization models and I notice that Lasso requires three inputs, a dependent variable and then two sets of what I assume as independent variables, one which are of interest and one from which the model selects.

What does this mean, exactly? I'm used to multiple regressions where you just insert the independent variables in one set.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

This distinction isn't always pre-specified in software, but it is often available (although this distinction might not be the default so that you have to read the manual to figure out how to do it). It's a way to limit regularization to predictors that control for additional influences on outcome while not penalizing coefficients of the predictors of primary interest.

The main point is that you don't need to penalize all the regression coefficients in a regularized model. That's true both for LASSO and for ridge regression.

For example, say that you have some predictors of primary interest and want to use other predictors primarily to control for additional influences on outcome. But you have too many predictors to incorporate without overfitting, based on the number of cases or events. In that situation you could choose not to penalize the predictors of primary interest and restrict penalization to coefficients of the predictors used to control for additional influences on outcome.

This paper illustrates the approach.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ (+1) nice explanation ! $\endgroup$
    – Robert Long
    Commented Jan 11, 2020 at 17:48
  • $\begingroup$ Thank you. But as regularization models are automated, assume I am interested in x1, x2, x3 and x4 because they seem to be correlated to my outcome but I willingly put only x1 into variables of interest, and the rest into the "picks from", wouldn't Lasso find that x2, x3 and x4 are also strongly correlated and keep them in the model anyhow? Just want to see if I understand this correctly. $\endgroup$
    – Paze
    Commented Jan 11, 2020 at 18:09
  • $\begingroup$ @Paze if x2, x3, and x4 are highly correlated with each other and with x1, then it's quite possible that they would not be selected at all, or maybe only 1 of them. The selections from the "picks from" set are effectively made by how much they improve performance beyond what the unpenalized variables of interest provide. When LASSO regularization is presented with several correlated predictors it tends to choose only 1 or a few from among them. In any case, the coefficients for variables selected from the "picks from" set would be penalized, unlike those for the variables of interest. $\endgroup$
    – EdM
    Commented Jan 11, 2020 at 21:18
  • $\begingroup$ That sounds complete opposite of how I understand multiple regression basics. If x2, x3 and x4 are correlated with each other and x1 (and y), this means they are confounders that should be corrected for by being included in the model? Or am I misunderstanding something? $\endgroup$
    – Paze
    Commented Jan 11, 2020 at 21:34
  • $\begingroup$ @Paze With LASSO you deliberately throw away predictors that do not improve model performance enough to warrant adding their coefficient magnitudes into the penalization term. The 1 or few variables kept in the model from a set of correlated predictors serve as proxies for the ones that were omitted. That is different by design from standard multiple regression: LASSO is a variable-selection method. Elastic net and ridge regression provide behavior more like what you expect. (I will admit to an aesthetic preference for ridge on that basis if there aren't very many predictors involved.) $\endgroup$
    – EdM
    Commented Jan 11, 2020 at 21:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.