5
$\begingroup$

My question is simple: when glmnet use alpha between 0 and 1 (i.e. elastic net), is it returning naive elastic net or adjusted one? Especially, for every method inbuilt (coef, cv.glmnet, predict) is it naive or adjusted?

The reason I am wondering about this is because I cannot align glmnet with enet.

require(ISLR)
library(glmnet)
library(elasticnet)
x=model.matrix(Apps~.,College)[,-1]
y=College$Apps
enet.model=enet(x,y,lambda=1)
predict(enet.model,type="coef",s=2,mode="penalty")
glmnet.model=glmnet(x,y,alpha=0.5)
coef(glmnet.model,s=4,exact=TRUE)

They give different results, why?

$\endgroup$
  • 3
    $\begingroup$ Is there any reason not to share the results with us? It might make it easier for people to see what is going on. $\endgroup$ – mdewey Oct 12 '16 at 9:17
  • $\begingroup$ Sorry I will be doing right now $\endgroup$ – Yuan Tao Oct 12 '16 at 11:53
3
$\begingroup$
x=model.matrix(Apps~.,College)[,-1]
y=College$Apps

enet.model=enet(x, y, lambda=1)
mx_x <- data.frame(coef_enet = round(predict(enet.model, type="coef", s=2, mode="penalty", naive = F)$coefficients, 2))
mx_x$coef_enet.naive <- round(predict(enet.model, type="coef", s=2, mode="penalty", naive = T)$coefficients, 2)

glmnet.model=glmnet(x, y, alpha=0.50)
mx_x$coef_glmnet <- round(coef(glmnet.model, s=2, exact=TRUE), 2)[-1]

mx_x

The script above gives:

               coef_enet coef_enet.naive coef_glmnet

PrivateYes   -1125.13         -562.56     -493.72
Accept           0.85            0.43        1.57
Enroll           1.44            0.72       -0.80
Top10perc       26.13           13.06       48.90
Top25perc       17.20            8.60      -13.47
F.Undergrad      0.25            0.12        0.05
P.Undergrad      0.21            0.10        0.04
Outstate         0.02            0.01       -0.08
Room.Board       0.31            0.15        0.15
Books            0.70            0.35        0.02
Personal         0.12            0.06        0.03
PhD             11.77            5.88       -8.53
Terminal        10.30            5.15       -3.31
S.F.Ratio       23.87           11.93       14.97
perc.alumni    -17.08           -8.54       -0.01
Expend           0.09            0.05        0.08
Grad.Rate       19.05            9.52        8.51

Using naive=FALSE for ElasticNet is just transforming the coefficients of naive=TRUE according to formula: coef(ENet) = (1 + lambda) * coef(NaiveENet)

The glmnet gives "ready-to-use" coefficients. However they are different from those of ElasticNet. (maybe different algorithms)

You'll get ElasticNet and glmnet "aligned" if you don't normalize the predictors.

enet.model=enet(x, y, lambda=1, normalize = F)
mx_x <- data.frame(coef_enet = round(predict(enet.model, type="coef", s=2,     mode="penalty", naive = F)$coefficients, 2))
mx_x$coef_enet.naive <- round(predict(enet.model, type="coef", s=2, mode="penalty", naive = T)$coefficients, 2)

glmnet.model=glmnet(x, y, alpha=0.50, standardize = F)
mx_x$coef_glmnet <- round(coef(glmnet.model, s=2, exact=TRUE), 2)[-1]

mx_x



        coef_enet coef_enet.naive coef_glmnet
PrivateYes    -971.26         -485.63     -481.73
Accept           3.17            1.59        1.58
Enroll          -1.76           -0.88       -0.87
Top10perc       99.84           49.92       49.57
Top25perc      -28.47          -14.24      -14.03
F.Undergrad      0.12            0.06        0.06
P.Undergrad      0.09            0.04        0.04
Outstate        -0.17           -0.09       -0.09
Room.Board       0.30            0.15        0.15
Books            0.04            0.02        0.02
Personal         0.06            0.03        0.03
PhD            -17.29           -8.65       -8.54
Terminal        -6.59           -3.30       -3.35
S.F.Ratio       31.04           15.52       15.48
perc.alumni      0.31            0.16        0.09
Expend           0.16            0.08        0.08
Grad.Rate       17.31            8.66        8.63

How predict works?

mx_x2 <- data.frame(pred_enet = predict(enet.model, newx=x, s=2, mode="penalty", naive=F)$fit,
pred_enet.naive = predict(enet.model, newx=x, s=2, mode="penalty", naive=T)$fit,
pred_glmnet = as.vector(predict(glmnet.model, newx=x, s=2, exact=T)))
head(mx_x2)

                          pred_enet pred_enet.naive pred_glmnet
Abilene Christian University  -200.7408       1400.4488   1403.1934
Adelphi University            3757.6684       3379.6534   3376.7797
Adrian College                -456.4379       1272.6002   1272.1445
Agnes Scott College            994.5582       1998.0983   1997.6611
Alaska Pacific University    -3453.9529       -226.1573   -221.7383
Albertson College            -1664.2285        668.7049    668.3047

enet prediction is linear in enet naive prediction (formula):

fit.lm <- lm(pred_enet~pred_enet.naive, data=mx_x2)
coef(fit.lm)
(Intercept) pred_enet.naive 
  -3001.638           2.000 
summary(fit.lm)$r.squared
[1] 1

enet naive is almost identical to glmnet prediction:

fit.lm <- lm(pred_enet.naive~-1+pred_glmnet, data=mx_x2)
coef(fit.lm)
pred_glmnet 
1.000126
summary(fit.lm)$r.squared
[1] 0.9999994
$\endgroup$
  • $\begingroup$ Thanks for the answer! I am still having following questions: $\endgroup$ – Yuan Tao Oct 12 '16 at 21:32
  • $\begingroup$ Why you use s=2 for glmnet? Isn't lambda=4 letting them having the same penalties? Also did predict and cv method in glmnet use naive coefficients? $\endgroup$ – Yuan Tao Oct 12 '16 at 21:34
  • $\begingroup$ @YuanTao "Why you use s=2 for glmnet? Isn't lambda=4 letting them having the same penalties?" I can't figure this out from authors' materials: elasticnet: users.stat.umn.edu/~zouxx019/Papers/elasticnet.pdf glmnet: web.stanford.edu/~hastie/glmnet/glmnet_alpha.html so I just assumed is the same or close to the same penalty when using s=2. To have exactly the same penalties, I think, alpha should be also changed. I have edited the answer for the second part of your question. $\endgroup$ – nvicol Oct 13 '16 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.