LASSO for explanatory models: shrinked parameters or not?

Question

I'm conducting an analysis where the primary goal is to understand the data. The dataset is large enough for cross-validation (10k), and predictors include both continuous and dummy variables, and the outcome is continuous. Main goal was to see if it makes sense to kick out some predictors, in order to make the model easier to interpret.

Questions:

1. My question is "which vars explain the outcome and are a 'strong enough' part of that explanation". But to select the lambda parameter for lasso, you use cross-validation, i.e, predictive validity as the criterion. When doing inference, is predictive validity a good enough proxy for the general question I am asking?

2. Say LASSO kept only 3 out of 8 predictors. And now I ask myself: "what effect do these have on the outcome". For example, I found a gender difference. After the lasso shrinkage, the coefficient suggests that women score 1 point higher than men. But without the shrinkage (i.e., on the actual dataset), they score 2.5 points higher.

   • Which one would I take as my "real" gender effect? Going only by predictive validity, it would be the shrunk coefficient.
   • Or in a context, say that I'm writing a report for people not well versed in statistics. Which coefficient would I report to them?

Answer 1

If your goal is to accurately estimate the parameters in your model then how close you are to the true model is how you should select your model. Predictive validity via cross-validation is one way to do this and is the preferred$^*$ way for selecting $\lambda$ in LASSO regression.

Now, to answer the question as to which parameter estimate is the "real estimate" one should look at which parameter is "closest" to the real parameter value. Does "closest" mean the parameter estimates that minimize bias? If so, then the least square estimator is unbiased in linear regression. Does closest mean the parameter estimate that minimizes mean square error (MSE)? Then it can be shown that there is a specification of ridge regression that will give you estimates that minimize MSE (similar to LASSO, ridge regression shrinks parameter estimates toward zero but, different from LASSO, parameter estimates do not reach zero). Similarly, there are several specifications of the tuning paramater $\lambda$ in LASSO that will result in smaller MSE than linear regression (see here). As the statistician, you have to determine what is the "best" estimate and report it (preferably with some indication of the confidence of the estimate) to those who are not well versed in statistics. What is "best" may or may not be a biased estimate.

The glmnet function in R does a pretty good job of selecting good values of $\lambda$ and, in summary, selecting $\lambda$ through cross-validation and reporting the parameter estimates is a perfectly reasonable way to estimate the "real" value of the parameters.

$^*$A Bayesian LASSO model that selects $\lambda$ by marginal likelihood is preferred by some but I'm, perhaps incorrectly, assuming you are doing a frequentist LASSO model.