Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I realize that when I use LASSO to select a model, I can't infer from it because it penalizes the coefficients to best fit the data.
I'm just a little confused as to what exactly this means. Isn't this also what a regression does? It tries to fit the data best possible?
If I can't infer from a LASSO model, what do I use it for?
I realize that when I use LASSO to select a model, I can't infer from it because it penalizes the coefficients to best fit the data. I'm just a little confused as to what exactly this means. Isn't this also what a regression does? It tries to fit the data best possible? If I can't infer from a LASSO model, what do I use it for?
LASSO is like OLS plus penalization. It was developed for prediction scope (out of sample) and not for inference or fitting. If the data are give, OLS is the best for fitting among all linear estimators. Moreover, at least in standard cases, OLS is better that LASSO for inference. The last because OLS is an unbiased estimator while LASSO are not.
In his basic form the problem is as follow. Suppose that we have a true model like this:
$y = X’\beta + \epsilon$
where $X$ is a vector of variables, $\beta$ a vector of coefficients, and exogeneity condition hold $E[\epsilon|X]=0$.
Here something about what I mean with true model and exogeneity condition (What is a 'true' model? ; What is the actual definition of endogeneity? ; endogenous regressor and correlation).
Now, if we perform an OLS regression like
$y = X’\theta +\ u$
we have that $\theta$ is an unbiased estimator of $\beta$. For simplicity we can intend unbiasedness and consistency as synonyms here.
While if we estimate, on the same data, a LASSO regression like
$y = X’\eta +\ v$
we have that, in general, $\theta \neq \eta$ then $\eta$ cannot be an unbiased estimator for $\beta$. This happen because the LASSO penalization imply that some terms of $\eta$ are shrunk towards zero.
Usually, or better historically, we was interested in unbiased (or consistent) estimators and for this reason estimators like LASSO do not had any role in many econometric books. However if we are focused on pure prediction LASSO, and other estimators, can become useful. This fact come from bias-variance tradeoff (test MSE minimization). These discussion can help (What is the relationship between minimizing prediciton error versus parameter estimation error? ; Is the idea of a bias-variance "tradeoff" a false construct? ).
All above reply to your main question, I hope. However from recent literature seems that estimator like LASSO can be useful also for inference, at least in high dimensional situation and/or using LASSO in first step and OLS in the second (read here: Inference after using Lasso for variable selection and LASSO Regression - p-values and coefficients ).
LASSO is a procedure that pushes some of the parameters which have small effect to be zero. It can thus be used to select the best subset of variables to predict your response.
You can eg use these variables in a least square fit afterwards, insteaf of the full set of parameters
