12
$\begingroup$

LASSO and adaptive LASSO are two different things, right? (To me, the penalties look different, but I'm just checking whether I miss something.)

When you generally speak about elastic net, is the special case LASSO or adaptive LASSO?

Which one does the glmnet package do, provided you choose alpha=1?

Adaptive LASSO works on milder conditions, right? Both have the oracle property in suitable data, right?

$\endgroup$
15
$\begingroup$

Brief answers to your questions:

  • Lasso and adaptive lasso are different. (Check Zou (2006) to see how adaptive lasso differs from standard lasso.)
  • Lasso is a special case of elastic net. (See Zou & Hastie (2005).)
    Adaptive lasso is not a special case of elastic net.
    Elastic net is not a special case of lasso or adaptive lasso.
  • Function glmnet in "glmnet" package in R performs lasso (not adaptive lasso) for alpha=1.
  • Does lasso work under milder conditions than adaptive lasso? I cannot answer this one (should check Zou (2006) for insights).
  • Only the adaptive lasso (but not lasso or elastic net) has the oracle property. (See Zou (2006).)

References:

| cite | improve this answer | |
$\endgroup$
12
$\begingroup$

LASSO solutions are solutions that minimize

$$Q(\beta|X,y) = \dfrac{1}{2n}||y-X\beta||^2 + \lambda\sum_{j}|\beta_j|$$

the adaptive lasso simply adds weights to this to try to counteract the known issue of LASSO estimates being biased.

$$Q_a(\beta|X,y,w) = \dfrac{1}{2n}||y-X\beta||^2 + \lambda\sum_{j}w_j|\beta_j|$$

Often you will see $w_j = 1/\tilde{\beta}_j$, where $\tilde{\beta}_j$ are some initial estimates of the $\beta$ (maybe from just using LASSO, or using least squares, etc). Sometimes adaptive lasso is fit using a "pathwise approach" where the weight is allowed to change with $\lambda$

$$w_j(\lambda) = w(\tilde{\beta}_j(\lambda))$$. In the $\texttt{glmnet}$ package the weights can be specified with the $\texttt{penalty.factor}$ argument. I'm not sure if you can specify the "pathwise approach" in $\texttt{glmnet}$.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ You forgot to take absolute values in the penalty terms. $\endgroup$ – Richard Hardy Aug 25 '16 at 11:13
  • $\begingroup$ Small addition: The denominator of the weights should be the $| \beta |^{\gamma}$, $\gamma$ is often set equal to 1, but could also be estimated using cross validation. Furthermore, it is only valid to use the $\beta$ obtained from a root-n consistent estimator (but you're right, LASSO and LS can be used). $\endgroup$ – Marcel10 Aug 25 '16 at 11:28
  • $\begingroup$ So basically, glmnet performs LASSO or elastic net by default, but you can switch this to adaptive LASSO (or EN) by specifying appropriate weights? If this is the case, thanks a million! $\endgroup$ – Mr Validation Aug 25 '16 at 11:59
  • $\begingroup$ @MrValidation, note that authors of new methods like adaptive lasso may have code for the method on their websites (sometimes they just give a reference to an R package that they themselves have written). $\endgroup$ – Richard Hardy Aug 30 '16 at 15:18
  • 1
    $\begingroup$ I think the weights argument in glmnet refers to weights for observations, and not weights for penalties $\endgroup$ – jmb Jan 24 '17 at 18:55
3
$\begingroup$

Adaptive LASSO is used for consistent variable selection. The problems we encounter when using the LASSO for variable selection are:

  • The shrinkage parameter must be larger for selection than prediction
  • Large nonzero parameters will be too small so that the bias is too large
  • Small nonzero parameters cannot be detected consistently
  • High correlations between predictors leads to poor selection performance

Thus the LASSO is only consistent for variable selection under some conditions on the shrinkage parameter, parameters (beta-min condition) and correlations (irrepresentable condition). See pages 101-106 of my masters dissertation for a detailed explanation.

The LASSO often includes too many variables when selecting the tuning parameter for prediction but the true model is very likely a subset of these variables. This suggests using a secondary stage of estimation like the adaptive LASSO which controls the bias of the LASSO estimate using the prediction-optimal tuning parameter. This leads to consistent selection (or oracle property) without the conditions mentioned above.

You can use glmnet for adaptive LASSO. First you need an initial estimate, either least squares, ridge or even LASSO estimates, to calculate weights. Then you can implement adaptive LASSO by scaling the X matrix. Here's an example using least squares initial estimates on training data:

# get data
y <- train[, 11]
x <- train[, -11]
x <- as.matrix(x)
n <- nrow(x)

# standardize data
ymean <- mean(y)
y <- y-mean(y)  
xmean <- colMeans(x)
xnorm <- sqrt(n-1)*apply(x,2,sd)
x <- scale(x, center = xmean, scale = xnorm)

# fit ols 
lm.fit <- lm(y ~ x)
beta.init <- coef(lm.fit)[-1] # exclude 0 intercept

# calculate weights
w  <- abs(beta.init)  
x2 <- scale(x, center=FALSE, scale=1/w)  

# fit adaptive lasso
require(glmnet)
lasso.fit <- cv.glmnet(x2, y, family = "gaussian", alpha = 1, standardize = FALSE, nfolds = 10)
beta <- predict(lasso.fit, x2, type="coefficients", s="lambda.min")[-1]

# calculate estimates
beta <- beta * w / xnorm # back to original scale
beta <- matrix(beta, nrow=1)
xmean <- matrix(xmean, nrow=10)
b0 <- apply(beta, 1, function(a) ymean - a %*% xmean) # intercept
coef <- cbind(b0, beta)
| cite | improve this answer | |
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.