How to estimate shrinkage parameter in Lasso or ridge regression?

Question

I want to use Lasso or ridge regression for a model with more than 50,000 variables. I want do so using software package in R. How can I estimate shrinkage parameter ($\lambda$)? Excuse me for the simple question as I new to the field...

Edits:

Here is upto the point I got to:

set.seed (123)
Y <- runif (1000)
Xv <- sample(c(1,0), size= 1000*1000,  replace = T)
X <- matrix(Xv, nrow = 1000, ncol = 1000)

mydf <- data.frame(Y, X)

require(MASS)
lm.ridge(Y ~ ., mydf)

plot(lm.ridge(Y ~ ., mydf,
              lambda = seq(0,0.1,0.001)))

My question is how do I know which $\lambda$ is best for my model ???

Answer 1

The function cv.glmnet from the R package glmnet does automatic cross-validation on a grid of $\lambda$ values used for $\ell_1$-penalized regression problems. In particular, for the lasso. The glmnet package also supports the more general elastic net penalty, which is a combination of $\ell_1$ and $\ell_2$ penalization. As of version 1.7.3. of the package taking the $\alpha$ parameter equal to 0 gives ridge regression (at least, this functionality was not documented until recently).

Cross-validation is an estimate of the expected generalization error for each $\lambda$ and $\lambda$ can sensibly be chosen as the minimizer of this estimate. The cv.glmnet function returns two values of $\lambda$. The minimizer, lambda.min, and the always larger lambda.1se, which is a heuristic choice of $\lambda$ producing a less complex model, for which the performance in terms of estimated expected generalization error is within one standard error of the minimum. Different choices of loss functions for measuring the generalization error are possible in the glmnet package. The argument type.measure specifies the loss function.

Alternatively, the R package mgcv contains extensive possibilities for estimation with quadratic penalization including automatic selection of the penalty parameters. Methods implemented include generalized cross-validation and REML, as mentioned in a comment. More details can be found in the package authors book: Wood, S.N. (2006) Generalized Additive Models: an introduction with R, CRC.

Answer 2

This answer is MATLAB specific, however, the basic concepts should be quite similar to what you're used to with R...

In the case of MATLAB, you have the option to run lasso with cross validation enabled.

If you do so, the lasso function will report two critical parameter values

1. The lambda value that minimizes the cross validated mean squared error
2. The lambda value with the greatest amount of shrinkage whose CVMSE is within one standard error of the minimum.

You also get a nice little chart that you can use to inspect the relationship between lambda and CVMSE

In general, you'll chose a value of lambda that falls between the blue line and the green line.

The following blog posting includes some demo code based on some examples in

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. Royal. Statist. Soc B., Vol. 58, No. 1, pages 267-288).

http://blogs.mathworks.com/loren/2011/11/29/subset-selection-and-regularization-part-2/

Answer 3

I have had good success using effective AIC, that is using AIC with the effective degrees of freedom - see Gray JASA 87:942 1992 for effective d.f. This is implemented for $L_{2}$ penalty in the R rms package for linear and logistic models, and the rms pentrace function can be used to solve for the shrinkage coefficient that optimizes effective AIC. A case study that shows how to do differential shrinkage (e.g. more shrinkage for interactions) is Harrell et al Stat in Med 17:909, 1998.