Lasso vs. adaptive Lasso

Question

LASSO and adaptive LASSO are two different things, right? (To me, the penalties look different, but I'm just checking whether I miss something.)

When you generally speak about elastic net, is the special case LASSO or adaptive LASSO?

Which one does the glmnet package do, provided you choose alpha=1?

Adaptive LASSO works on milder conditions, right? Both have the oracle property in suitable data, right?

Answer 1

Brief answers to your questions:

• Lasso and adaptive lasso are different. (Check Zou (2006) to see how adaptive lasso differs from standard lasso.)
• Lasso is a special case of elastic net. (See Zou & Hastie (2005).)
  Adaptive lasso is not a special case of elastic net.
  Elastic net is not a special case of lasso or adaptive lasso.
• Function glmnet in "glmnet" package in R performs lasso (not adaptive lasso) for alpha=1.
• Does lasso work under milder conditions than adaptive lasso? I cannot answer this one (should check Zou (2006) for insights).
• Only the adaptive lasso (but not lasso or elastic net) has the oracle property. (See Zou (2006).)

References:

• Zou, Hui. "The adaptive lasso and its oracle properties." Journal of the American Statistical Association 101.476 (2006): 1418-1429.
• Zou, Hui, and Trevor Hastie. "Regularization and variable selection via the elastic net." Journal of the Royal Statistical Society: Series B (Statistical Methodology) 67.2 (2005): 301-320.

Answer 2

LASSO solutions are solutions that minimize

$$Q(\beta|X,y) = \dfrac{1}{2n}||y-X\beta||^2 + \lambda\sum_{j}|\beta_j|$$

the adaptive lasso simply adds weights to this to try to counteract the known issue of LASSO estimates being biased.

$$Q_a(\beta|X,y,w) = \dfrac{1}{2n}||y-X\beta||^2 + \lambda\sum_{j}w_j|\beta_j|$$

Often you will see $w_j = 1/\tilde{\beta}_j$, where $\tilde{\beta}_j$ are some initial estimates of the $\beta$ (maybe from just using LASSO, or using least squares, etc). Sometimes adaptive lasso is fit using a "pathwise approach" where the weight is allowed to change with $\lambda$

$$w_j(\lambda) = w(\tilde{\beta}_j(\lambda))$$. In the $\texttt{glmnet}$ package the weights can be specified with the $\texttt{penalty.factor}$ argument. I'm not sure if you can specify the "pathwise approach" in $\texttt{glmnet}$.

Answer 3

Adaptive LASSO is used for consistent variable selection. The problems we encounter when using the LASSO for variable selection are:

• The shrinkage parameter must be larger for selection than prediction
• Large nonzero parameters will be too small so that the bias is too large
• Small nonzero parameters cannot be detected consistently
• High correlations between predictors leads to poor selection performance

Thus the LASSO is only consistent for variable selection under some conditions on the shrinkage parameter, parameters (beta-min condition) and correlations (irrepresentable condition). See pages 101-106 of my masters dissertation for a detailed explanation.

The LASSO often includes too many variables when selecting the tuning parameter for prediction but the true model is very likely a subset of these variables. This suggests using a secondary stage of estimation like the adaptive LASSO which controls the bias of the LASSO estimate using the prediction-optimal tuning parameter. This leads to consistent selection (or oracle property) without the conditions mentioned above.

You can use glmnet for adaptive LASSO. First you need an initial estimate, either least squares, ridge or even LASSO estimates, to calculate weights. Then you can implement adaptive LASSO by scaling the X matrix. Here's an example using least squares initial estimates on training data:

# get data
y <- train[, 11]
x <- train[, -11]
x <- as.matrix(x)
n <- nrow(x)

# standardize data
ymean <- mean(y)
y <- y-mean(y)  
xmean <- colMeans(x)
xnorm <- sqrt(n-1)*apply(x,2,sd)
x <- scale(x, center = xmean, scale = xnorm)

# fit ols 
lm.fit <- lm(y ~ x)
beta.init <- coef(lm.fit)[-1] # exclude 0 intercept

# calculate weights
w  <- abs(beta.init)  
x2 <- scale(x, center=FALSE, scale=1/w)  

# fit adaptive lasso
require(glmnet)
lasso.fit <- cv.glmnet(x2, y, family = "gaussian", alpha = 1, standardize = FALSE, nfolds = 10)
beta <- predict(lasso.fit, x2, type="coefficients", s="lambda.min")[-1]

# calculate estimates
beta <- beta * w / xnorm # back to original scale
beta <- matrix(beta, nrow=1)
xmean <- matrix(xmean, nrow=10)
b0 <- apply(beta, 1, function(a) ymean - a %*% xmean) # intercept
coef <- cbind(b0, beta)