2
$\begingroup$

For research purposes I'm trying to predict the transfer prices of football players. I've implemented several models of which one is Lasso. Now, besides prediction, I would like to measure actual effects of certain variables, for which I've implemented Post-Lasso estimation (Least squares on the non-zero coefficients chosen by Lasso). I've read on several sites, and for one here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4207812/ that the standard errors are wrong, because they do not take into account the selection procedure which I've used before the least squares estimation. Furthermore, I am dealing with heteroskedasticity and therefore I cannot use the 'normal' standard errors returned by OLS.

Now I was wondering whether or not the bootstrap method: Bagging [1] works to deal with this problem. I would sample with replacements multiple times, run least squares on all these samples and calculate the standard errors over all the coefficients.

My question: Would this work and why (not)?

Thanks a lot!

[1] Breiman, L. (1996). Bagging predictors. Machine learning, 24(2), 123-140.

$\endgroup$
  • $\begingroup$ You could use a standard non parametric bootstrap. However, I do not see that standard errors are very useful for penalized regression. You are intentionally inducing bias into the model, such that variance is reduced. In order to ontain a variance estimate, you would need to know to the bias. Outside of simmulations, I cant see that this will work $\endgroup$ – Repmat Sep 13 '17 at 7:17
  • $\begingroup$ To reply to @repmat his answer. I'm not trying to estimate the standard errors on my Penalized regression. I'm using the Lasso solely for prediction. I've introduced the Least squares simply to give information regarding the variables that Lasso includes, and what their true values are (for understanding purposes, not prediction). For this reason I need to adjust the standard errors on the least squares estimates, to take into account that I've used a model selection before using the least squares estimation. Regarding your standard non parametric bootstrap. I'm unfamiliar with this technique. $\endgroup$ – Ivo Hendriks Sep 13 '17 at 7:28
  • $\begingroup$ @IvoHendriks please do not use answers for commenting questions. Answers are meant for answering them. Also please notice that if you have multiple accounts, you can merge them. $\endgroup$ – Tim Sep 13 '17 at 7:34
  • $\begingroup$ @IvoHendriks that doesnt help. You have biased and inconsistent parameter estimates, inference is essentially worthless in this case. $\endgroup$ – Repmat Sep 13 '17 at 8:08
2
$\begingroup$

What you propose is to select variables via LASSO first, then use the selected variables in a second unpenalized regression model to try to estimate their "actual effects." The comments from @Repmat would certainly apply if you were working directly with the LASSO-derived coefficients; coefficients from LASSO are deliberately biased. Your approach appears to be an attempt to remove that bias.

Yes, there are ways to estimate standard errors in this type of approach, but they have to take into account the pre-selection and the discontinuities introduced by the variable-selection process. The 2014 paper by Efron that you cite provides one such approach. Efron's method, however, is more than the simple bootstrap that you propose. It repeats the model selection process over the set of bootstrap samples, while your proposal limited to your initial LASSO-based selection takes neither the pre-selection nor the associated discontinuities into account. Note that Efron introduced the bootstrap in a 1977 lecture (The Annals of Statistics 7:1-26, 1979; doi: 10.1214/aos/1176344552), and it took him 35 more years to come up with a principled way to use it to estimate standard errors in such variable-selection schemes.

There is no reason that you could not simply follow Efron's 2014 method, if you wish; the non-parametric approaches seem like they should work with underlying heteroskedasticity. (Parametric bootstraps typically require a well-specified model.) Related references involving OLS following LASSO can be found linked from this page.

What I fear in such approaches is omitted-variable bias. In addition to its Wikipedia page, a search on this site today provides 307 results. If you omit important predictors from a model, the coefficients of the remaining coefficients may be biased from their "actual effects." LASSO tends to pick only a few of multiple correlated predictors, even if all of them have appreciable "actual effects." So the "actual effects" of the remaining variables in your regression-after-LASSO approach run the risk of having such omitted-variable bias. (Note that this omitted-variable bias is distinct from the coefficient bias inherently introduced by LASSO.)

A further caution is the variability of the set of selected variables among multiple samples. Try repeating the entire LASSO modeling approach on multiple bootstrap samples from your data (say, 200 or more) and see how different the selected sets of variables can be. (This is related to the "discontinuity" issue noted above.) In that case, which predictors have the "actual effects"? This is also a useful way to estimate the quality of your modeling process: for example, use each of the models derived from the bootstrapped samples to predict the transfer prices in the original data set to estimate the standard errors of the predictions. With your interest in predictions, those estimates of prediction errors are likely to be much more useful than any estimates of the errors in the coefficients themselves.

Note that the basic approach in the Efron 2014 paper has some similarity to my recommendation in the previous paragraph: the approach is to smooth out the discontinuities from variable selection by averaging results over a set of models each based on a separate bootstrap sample. These models in general will differ in the number or identity of the predictors used. Also, Efron explicitly notes in his parametric LASSO example that "estimation of the regression coefficients [is] a more difficult task" than estimating a single statistic like your transfer prices (page 19 of the linked PMC manuscript, page 1001 of the published JASA article: v109,p991).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.