6
$\begingroup$

Is it reasonable to average LASSO coefficients from repeated reshuffling of training/test sets?

Suppose I randomly divide my data into testing & training sets, then within the training set use 10-fold cross-validation to choose an optimal $\lambda$, then refit on full training data and record the model coefficients. Now, suppose that I repeat this process $k$ number of times. Each iteration will choose slightly different coefficients. One might think to average these together. However, each iteration may not choose the same set of non-zero coefficients, therefore the average of all coefficients may contain many more non-zero coefficients than any single solution.

I found this similar brief discussion here, but I do not want to be confused by the additional discussion of multiple imputation: Combining LASSO coefficients across imputed datasets Please note that no answer was ever accepted for this question.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

A similar thing with bootstrap replication is implemented in the "bolasso" function of the R package "mht" (for multiple hypothesis testing), and published here http://www.di.ens.fr/sierra/pdfs/icml_bolasso.pdf but they take the intersection of the sets of predictors with nonzero coefficients from all the replication samples, and then fit unregularized least squares estimators using only those variables.

You pointed out the problem with taking the union of the supports, that you lose the advantage of dimensionality reduction, and your Lasso estimates are still biased.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks. Bolasso seems interesting for feature selection, but what if I still want some coefficient shrinkage instead of OLS? $\endgroup$
    – raco
    Commented Feb 18, 2014 at 20:21
  • 1
    $\begingroup$ Why would you want that? Maybe use ridge regression? $\endgroup$
    – vafisher
    Commented Feb 18, 2014 at 21:00
  • $\begingroup$ For the same reasons that regularized regression often outperforms least squares out of sample. I think that averaging ridge coefficients should work after applying some rule of thumb to the lasso results for feature selection (i.e., intersection, union, or some other threshold for min % of non-zero coefficient selection per feature) $\endgroup$
    – raco
    Commented Feb 20, 2014 at 19:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.