3
$\begingroup$

L1 regularisation works by applying a penalty that reduces the magnitude of the coefficients. Some coefficients are driven to zero while the others are reduced but remain non-zero.

The remaining non-zero coefficients will therefore be lower than their true values. By true, I mean that if you had only used those dimensions from the beginning and fit without regularization, the coefficient values would be higher and the overall fit more accurate than what you have now.

However, it seems common practise to simply use the underestimated coefficient values as they are. Why is this?

$\endgroup$
1
  • 2
    $\begingroup$ If you fit the model with just those remaining non-zero coefficients and no regularization, you will get different coefficient estimates. Think about LASSO as just another way to estimate the parameters. Some people estimate variance by $\hat{\sigma}^2 = \dfrac{1}{n-1}\sum (x_i - \bar{x})^2$, and some people estimate variance by $\hat{\sigma}^2 = \dfrac{1}{n+1}\sum (x_i - \bar{x})^2$ because they believe such as estimate to have superior properties. (In particular, the second one has a smaller MSE, assuming a Gaussian distribution.) It is the same idea with LASSO estimation of the parameters. $\endgroup$
    – Dave
    May 10 at 13:51
4
$\begingroup$

The characterization of the unbiased estimates as the "true" values is inaccurate, and I think this may be the origin of some of the confusion underlying the question. While it's true that the lasso coefficient estimates are biased, it's does not follow that the unbiased coefficients will correctly match the parameters of the data generation process, even if the model is correctly specified and all of the regression requirements are met. This caveat is true whether or not LASSO is used.

Additionally, the LASSO introduces its own caveats; it is an imperfect screen. The procedure could inaccurately include an irrelevant predictor or exclude a relevant predictor.

Using just the LASSO estimate, without refitting the model, implies that the researcher desires a particular trade of bias for variance. The LASSO estimates will be biased but perhaps exhibit lower variance. In the extreme case of an under-determined regression, the biased LASSO estimates admit estimation of model parameters even in the setting with more predictors than observations ($p\gg n$), but if we have reduced the model to merely $n$ predictors, the ML estimates for these parameters will have enormous variance.

That said, a few examples of similar "phased" pipelines exist. One is the "relaxed lasso", which applies lasso regression twice, once to down-select from a large group to a small group of features, and second to estimate coefficients for use in a model. This uses cross-validation at each step to choose the magnitude of the penalty. The reasoning is that in the first step, you cross-validate and will likely choose a large penalty to screen out irrelevant predictors; in the second step, you cross-validate and will likely pick a smaller penalty (and hence larger coefficients). This is mentioned briefly in Tibshirani et al., Elements of Statistical Learning (pages 91-92) with a citation to Nicolai Meinshausen ("Relaxed Lasso." Computational Statistics & Data Analysis Volume 52, Issue 1, 15 September 2007, pp 374-393). ESL also briefly mentions a LASSO-then-MLE pipeline, but does not include any particular citation.

$\endgroup$
6
  • 1
    $\begingroup$ The lasso-then-mle estimator is within the class of relaxed lasso estimators. In Meinshausen's work, he just stipulates that the tuning parameter of the second lasso is smaller, i.e. between 0 and the first stage's tuning parameter. $\endgroup$
    – user257566
    May 10 at 22:24
  • 1
    $\begingroup$ @user257566 Sure, if the second stage of CV selects 0 as the LASSO shrinkage parameter, then the model has MLE coefficients on the reduced set of features. But that is a special case which occurs only if the second stage LASSO selects 0. A LASSO-then-MLE pipeline would only use MLE at the second stage, instead of conducting a second round of CV to find the optimal shrinkage parameter. That's the distinction I'm making. $\endgroup$
    – Sycorax
    May 10 at 23:38
  • 1
    $\begingroup$ Hey, the relaxed lasso doesn't include CV to select the second parameter. That's something you can do and that's part of what the paper does, but that's not the definition of the method. (See Definition 1 of the method in Meinshausen's paper.) In fact, almost all active statisticians I've met, including Rob Tibshirani, call the "relaxed lasso" the method in the particular case that the tuning parameter is 0 at the second stage, see eg cran.r-project.org/web/packages/glmnet/vignettes/relax.pdf $\endgroup$
    – user257566
    May 11 at 0:14
  • 1
    $\begingroup$ @user257566 Hastie, Tibshirani and Friedman give the same account in Elements of Statistical Learning. "The idea is to use cross-validation to estimate the initial penalty parameter for the lasso, and then again for a second penalty parameter applied to the selected set of predictors." (page 91) The CRAN vignette that you cite also includes the parenthetical "We note that there have been other definitions of a relaxed fit, but this is the one we prefer," suggesting that this account is not definitive, but an expression of these authors' preferences. $\endgroup$
    – Sycorax
    May 11 at 0:17
  • 1
    $\begingroup$ Thanks for sharing that. This isn't a major point--I'm just commenting because this is highlighted at the end of your post. There seems to be some kind of misunderstanding, though. IMO, I think its not best to focus on a sentence in a textbook which is trying to communicate some high level intuition instead of considering all other sources, especially since the 3 sentences preceding that one in ESL give the actual definition that i'm highlighting. Just to repeat for emphasis: the paper you cite was written in part to study lasso-then-mle. $\endgroup$
    – user257566
    May 11 at 0:25
2
$\begingroup$

the coefficient values would be higher and the overall fit more accurate than what you have now"

Why do you think this holds? The bias-variance tradeoff still holds even for a selection of variables.

Unconstrained parameters may be unbiased (assuming we didn't lasso out any variables with nonzero true parameters, a heroic assumption), but they will have a given variance. So constraining them may increase their bias, but will also reduce their variance, and may overall yield a lower error.

$\endgroup$
1
$\begingroup$

If you fit the model with just those remaining non-zero coefficients and no regularization, you will get different coefficient estimates. Think about LASSO as just another way to estimate the parameters. Some people estimate variance by $\hat{\sigma}^2 = \dfrac{1}{n-1}\displaystyle\sum_{i=1}^{n} \big(x_i - \bar{x}\big)^2$, and some people estimate variance by $\hat{\sigma}^2 = \dfrac{1}{n+1}\displaystyle\sum_{i=1}^{n} \big(x_i - \bar{x}\big)^2$ because they believe such as estimate to have superior properties. (In particular, the second one has a smaller mean squared error, assuming a Gaussian distribution.) It is the same idea with LASSO estimation of the parameters; we like the LASSO estimate over the OLS estimate because we think we can get a lower mean squared error by trading in our unbiased OLS estimator for a biased LASSO estimator that has a lot less variance.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.