2
$\begingroup$

I realize that when I use LASSO to select a model, I can't infer from it because it penalizes the coefficients to best fit the data.

I'm just a little confused as to what exactly this means. Isn't this also what a regression does? It tries to fit the data best possible?

If I can't infer from a LASSO model, what do I use it for?

$\endgroup$
4
  • $\begingroup$ what does "infer" mean? $\endgroup$ – user257566 Oct 24 '20 at 20:13
  • $\begingroup$ See for instance here. There are a lot of questions regarding the issue with inference from Lasso. $\endgroup$ – treskov Oct 24 '20 at 20:37
  • $\begingroup$ Thanks, treskov. I'm wondering specifically about what OP has in mind, though $\endgroup$ – user257566 Oct 24 '20 at 20:51
  • $\begingroup$ This is what a statistician told me. I can use LASSO to filter variables, but I cannot infer from a LASSO model. $\endgroup$ – Paze Oct 24 '20 at 21:07
5
+50
$\begingroup$

I realize that when I use LASSO to select a model, I can't infer from it because it penalizes the coefficients to best fit the data. I'm just a little confused as to what exactly this means. Isn't this also what a regression does? It tries to fit the data best possible? If I can't infer from a LASSO model, what do I use it for?

LASSO is like OLS plus penalization. It was developed for prediction scope (out of sample) and not for inference or fitting. If the data are give, OLS is the best for fitting among all linear estimators. Moreover, at least in standard cases, OLS is better that LASSO for inference. The last because OLS is an unbiased estimator while LASSO are not.

In his basic form the problem is as follow. Suppose that we have a true model like this:

$y = X’\beta + \epsilon$

where $X$ is a vector of variables, $\beta$ a vector of coefficients, and exogeneity condition hold $E[\epsilon|X]=0$.

Here something about what I mean with true model and exogeneity condition (What is a 'true' model? ; What is the actual definition of endogeneity? ; endogenous regressor and correlation).

Now, if we perform an OLS regression like

$y = X’\theta +\ u$

we have that $\theta$ is an unbiased estimator of $\beta$. For simplicity we can intend unbiasedness and consistency as synonyms here.

While if we estimate, on the same data, a LASSO regression like

$y = X’\eta +\ v$

we have that, in general, $\theta \neq \eta$ then $\eta$ cannot be an unbiased estimator for $\beta$. This happen because the LASSO penalization imply that some terms of $\eta$ are shrunk towards zero.

Usually, or better historically, we was interested in unbiased (or consistent) estimators and for this reason estimators like LASSO do not had any role in many econometric books. However if we are focused on pure prediction LASSO, and other estimators, can become useful. This fact come from bias-variance tradeoff (test MSE minimization). These discussion can help (What is the relationship between minimizing prediciton error versus parameter estimation error? ; Is the idea of a bias-variance "tradeoff" a false construct? ).

All above reply to your main question, I hope. However from recent literature seems that estimator like LASSO can be useful also for inference, at least in high dimensional situation and/or using LASSO in first step and OLS in the second (read here: Inference after using Lasso for variable selection and LASSO Regression - p-values and coefficients ).

$\endgroup$
-1
$\begingroup$

LASSO is a procedure that pushes some of the parameters which have small effect to be zero. It can thus be used to select the best subset of variables to predict your response.

You can eg use these variables in a least square fit afterwards, insteaf of the full set of parameters

$\endgroup$
3
  • $\begingroup$ There is no guarantee that LASSO selects the "best" variables in terms of inference. It might in terms of prediction, but then, rather than say ridge regression, why bother with selection at all? $\endgroup$ – Frans Rodenburg Jan 26 '20 at 14:01
  • $\begingroup$ The reason why we want to perform variable selection in inference is that if we have a lower number of parameters to estimate, the parameters are estimated at a greater precision. (Standard errors will be smaller) $\endgroup$ – Abm Jan 26 '20 at 14:39
  • $\begingroup$ I understand that, I just don't understand why I can't use the OLS model derived from the LASSO to infer. $\endgroup$ – Paze Jan 26 '20 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.