Why can't regression via Maximum Likelihood shrink regression coefficients to zero as in LASSO? Does shrinking coefficients to zero not give higher L-likelihood? Does the answer to my question have to do with the fact LASSO is bayesian with Laplace priors and Maximum likelihood is frequentist?

  • 1
    $\begingroup$ That just isn't whole ML works. You can introduce a penalty (which is what Lasso does). I would also recommend one well thought out question instead of 3 small questions. I've answered two of your questions within the last hour. $\endgroup$ – Demetri Pananos Jan 16 at 18:11
  • $\begingroup$ I've ran regressions via ML and I've never had a coefficient via ML be zero so i just assumed ML doesn't let the coefficient be zero because letting it be zero doesn't maximize the log-likelihood. $\endgroup$ – Numbers Jan 16 at 18:17
  • $\begingroup$ Its an unconstrained maximization problem. You will never get coefficients which are exactly 0. $\endgroup$ – Demetri Pananos Jan 16 at 18:18

Maximum Likelihood just like minimizing the square of the sum of residuals has no penalty factor to reduce coefficients to zero as LASSO does. So by definition Maximum Likelihood does not shrink coefficients to zero and LASSO does.

However, you can't automatically deduce that the LASSO model will generate a better fit and prediction of what you are estimating. Very often it does not. There are several problems with LASSO.

First, given its penalty factor LASSO has no way of differentiating between a true causal variable that has a high coefficient and should be selected in your model vs. another variable that has little relationship with Y and has a low coefficient. The LASSO algorithm may randomly and often select the weak variable instead of the strong causal variable. And, that's a big problem.

The second problem is that LASSO can completely dismantle the underlying explanatory logic of your original model. Not only it eliminates at times causal variables and substitute them with weak variables (1st problem); it can even at times change the directional sign of your variable coefficients. That's another big problem. You can visualize this problem by doing a search for Images of LASSO coefficient paths. And, you will see how so many LASSO models have very unstable coefficient paths that truly do not make any sense.

The third problem is that LASSO does not automatically generates better forecasts (out-of-sample predictions). Very often it does not. You can visualize that by doing a search for Images of LASSO MSE graphs. These graphs have MSE on the Y-axis with penalty factor on the X-axis. And, many LASSO models out there show a rapidly rising MSE the minute the penalty factor increases. This means that the original model with a penalty factor of Zero was much better at predicting than any of the LASSO models. This situation is surprisingly frequent.

  • $\begingroup$ “This means that the original model with a penalty factor of Zero was much better at predicting than any of the LASSO models. This situation is surprisingly frequent.” I am very skeptical about this sentence and never happens with data I am working with. Maybe you can demonstrate with a small example. I am talking about out of sample error of course. $\endgroup$ – Cagdas Ozgenc Jan 16 at 20:11
  • $\begingroup$ @CagdasOzgenc This might only be when all predictors affect the outcome. Traditionally, trading variance for bias should result in better out of sample predictions. $\endgroup$ – Demetri Pananos Jan 16 at 20:37
  • $\begingroup$ Ok, I have hit on a bit of healthy skepticism. I am not going to convince anyone of my viewpoints. Instead, I'll let anyone's own eyeballs convince themselves on my behalf. Just do an Internet search for Images "LASSO MSE." And, you should readily observe plenty of LASSO models whose MSE increases the minute the penalty factor rises. In other words, the best model was the original one before LASSO regularization. There are a ton of examples out there. I conducted such an Image search on Google a minute ago. And, nearly half of the LASSO MSE graphs had the characteristics I describe. $\endgroup$ – Sympa Jan 16 at 22:48
  • $\begingroup$ I see the images but when I actually open those pages to see the details none of them had a 0 lambda value selected. The lambda was simply a small value. Are you sure you are talking about exactly 0 lambda? $\endgroup$ – Cagdas Ozgenc Jan 17 at 11:37
  • $\begingroup$ You describe a technicality. When you are dealing with log(lambda) of - 8 or - 10 at the beginning of the X-axis, these values are so close to Zero that they pretty much eliminate the penalty factor out of the LASSO algorithm. And, at that point you are just left with a standard OLS Regression (what I call your original model pre-regularization). $\endgroup$ – Sympa Jan 17 at 16:33

I'll add this: That the LASSO (and associated with it, ridge regression) have Bayesian interpretations is a glimmer of light in the problem of penalization and inference. However, it isn't as easy as "pick priors so that some covariates are shrunk to zero". In order for valid inferences to be made from a bayesian linear model, your priors have to actually reflect your present uncertainty about the phenomenon. Else, the uncertainty yielded in the posterior does not correspond to anything, and your just computing expectations for the sake of doing math.

There is no answer to "How do I select variables for inference". Inference presupposes you have a well defined question to assess. Without that question, what are you really trying to infer?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.