9
$\begingroup$

I'm conducting an analysis where the primary goal is to understand the data. The dataset is large enough for cross-validation (10k), and predictors include both continuous and dummy variables, and the outcome is continuous. Main goal was to see if it makes sense to kick out some predictors, in order to make the model easier to interpret.

Questions:

  1. My question is "which vars explain the outcome and are a 'strong enough' part of that explanation". But to select the lambda parameter for lasso, you use cross-validation, i.e, predictive validity as the criterion. When doing inference, is predictive validity a good enough proxy for the general question I am asking?

  2. Say LASSO kept only 3 out of 8 predictors. And now I ask myself: "what effect do these have on the outcome". For example, I found a gender difference. After the lasso shrinkage, the coefficient suggests that women score 1 point higher than men. But without the shrinkage (i.e., on the actual dataset), they score 2.5 points higher.

    • Which one would I take as my "real" gender effect? Going only by predictive validity, it would be the shrunk coefficient.
    • Or in a context, say that I'm writing a report for people not well versed in statistics. Which coefficient would I report to them?
$\endgroup$
  • 1
    $\begingroup$ What kind of model are you looking at? Linear, logistic, poisson, etc. model? $\endgroup$ – TrynnaDoStat Mar 29 '15 at 16:12
  • 1
    $\begingroup$ It's a linear model, but I don't think that makes a difference for the question $\endgroup$ – mbokulic Mar 29 '15 at 17:04
7
$\begingroup$

If your goal is to accurately estimate the parameters in your model then how close you are to the true model is how you should select your model. Predictive validity via cross-validation is one way to do this and is the preferred$^*$ way for selecting $\lambda$ in LASSO regression.

Now, to answer the question as to which parameter estimate is the "real estimate" one should look at which parameter is "closest" to the real parameter value. Does "closest" mean the parameter estimates that minimize bias? If so, then the least square estimator is unbiased in linear regression. Does closest mean the parameter estimate that minimizes mean square error (MSE)? Then it can be shown that there is a specification of ridge regression that will give you estimates that minimize MSE (similar to LASSO, ridge regression shrinks parameter estimates toward zero but, different from LASSO, parameter estimates do not reach zero). Similarly, there are several specifications of the tuning paramater $\lambda$ in LASSO that will result in smaller MSE than linear regression (see here). As the statistician, you have to determine what is the "best" estimate and report it (preferably with some indication of the confidence of the estimate) to those who are not well versed in statistics. What is "best" may or may not be a biased estimate.

The glmnet function in R does a pretty good job of selecting good values of $\lambda$ and, in summary, selecting $\lambda$ through cross-validation and reporting the parameter estimates is a perfectly reasonable way to estimate the "real" value of the parameters.

$^*$A Bayesian LASSO model that selects $\lambda$ by marginal likelihood is preferred by some but I'm, perhaps incorrectly, assuming you are doing a frequentist LASSO model.

$\endgroup$
  • $\begingroup$ What did you mean by "bias" in "the parameter estimates that minimize bias"? And do I read the rest correctly if I read it like this: I should choose the model that has the lowest out-of-sample estimated MSE (i.e., in cross-validation)? Since ridge is out of the question since I want a sparse coefficient matrix, reporting the shrinked lasso coefficients is the way to go $\endgroup$ – mbokulic Apr 6 '15 at 19:07
  • $\begingroup$ @mbokulic By bias I mean statistical bias. This refers to the tendency of a measurement process to over/under estimate the value of a population parameter. My answer is saying that it depends what you want. If you don't want bias, stick with linear regression. If you're okay with bias and prefer to minimize MSE, go with LASSO and put in the due diligence when selecting $\lambda$. $\endgroup$ – TrynnaDoStat Apr 6 '15 at 19:26
  • $\begingroup$ interesting, I never thought of it in that way. Again I have to ask if I understood you correctly. So linear regression gives you the most unbiased estimate of the population coefficients (the "2.5 points higher" example in my original question). Whereas lasso or ridge regr. minimize out-of-sample MSE. If so, if you just want to understand (not predict), linear regression seems better, though you'd still want to simplify the model with e.g., stepwise methods. $\endgroup$ – mbokulic Apr 8 '15 at 18:28
  • $\begingroup$ the answers here are helpful. They suggest that OLS (linear regression) have the bast in-sample performance, whereas lasso is for out-of-sample. Also, they suggest that OLS can be used on the restricted set of predictors picked out by the lasso. This is exactly what makes sense for my interpretational goal, even if the OLS estimates will slightly overfit. $\endgroup$ – mbokulic Jun 14 '15 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.