2
$\begingroup$

As far as I know, if I run a lasso model and a ridge model on the same data, and if i keep lambda=0, I'm getting the OLS. Then, how is it possible that I get different results?

rm(list=ls())
library(caret)

x1=rnorm(100, mean = 0, sd = 1)
x2=rnorm(100, mean = 3, sd = 1)
y=x1+x2+rnorm(100, mean = 0.5, sd = 1)
dat=cbind(x1,x2,y)

control=trainControl(method = "cv",number=5)

set.seed(849)
ridge_caret<- train(dat[,1:2],dat[,3], method = "glmnet",
                trControl=control,preProc = c("center","scale"),
                tuneGrid = expand.grid(alpha = 0,
                                       lambda = 0))

set.seed(849)
lasso_caret<- train(dat[,1:2],dat[,3], method = "glmnet",
                trControl=control,preProc = c("center","scale"),
                tuneGrid = expand.grid(alpha = 1,
                                       lambda = 0))

For ridge, I'm getting:

RMSE      Rsquared   MAE      
1.031121  0.7159096  0.8503881

And for lasso,

RMSE      Rsquared   MAE      
1.031887  0.7157566  0.8485924

And of course, different coefficients:

> coef(ridge_caret$finalModel, ridge_caret$finalModel$lambdaOpt)
3 x 1 sparse Matrix of class "dgCMatrix"
                1
(Intercept) 3.6183601
x1          0.9203728
x2          0.9718673
> coef(lasso_caret$finalModel, lasso_caret$finalModel$lambdaOpt)
3 x 1 sparse Matrix of class "dgCMatrix"
                1
(Intercept) 3.6183601
x1          0.9657077
x2          1.0208237

¿Why are not they exactly the same?

$\endgroup$
  • 1
    $\begingroup$ You're not doing OLS. If you want to do that, don't cross-validate. With the cross-validation, you're doing a collection of OLS estimates on random subsets of the data. $\endgroup$ – whuber Aug 22 '18 at 13:24
  • 2
    $\begingroup$ Well, the fact is that I'm not really cross-validating to get my parameters, I'm just doing it to get an estimation of the test set error via RMSE... My parameters are fixed, and being lambda=0, I'm optimizing the same equations in both cases, so I expect the results to be the same. Caret returns the best cross-validated model, in this case the only one , trained on the whole set (lambda=0 so is OLS, right?), so I think that both RMSE...and parameters should be the same. $\endgroup$ – PeterTschuschke Aug 22 '18 at 16:34
  • 1
    $\begingroup$ This has nothing to do with cross-validation. Peter is right to question what is going on (+1). Given that the seed is fixed the bootstrapped/CV error should be exactly the same. $\endgroup$ – usεr11852 Aug 22 '18 at 19:53
6
$\begingroup$

I think that somewhat unfortunately you have hit a minor bug in caret's implementation of the glmnet model. (Bug in the sense of "unintended behaviour") Your understanding about how GLMNet works is correct.

What you expect, should happen (i.e. setting $\lambda = 0$ should result to exactly the same estimate irrespective of $\alpha$ values) but it does not happen because actually the glmnet code in caret will "ignore" the lambda values so it can "hot-start" the LASSO optimisation. (It then manually treats the lambda from the grid as the "optimal $\lambda$" lambdaOpt to use it for the predictions that will give the RMSE/MAE estimates.)

If you want to use $\lambda = 0$, I would recommend you explicitly set it as an additional argument, like:

set.seed(849)
ridge_caret<- train(dat[,1:2],dat[,3], method = "glmnet",lambda= 0,
                    tuneGrid = expand.grid(alpha = 0, lambda = 0))

set.seed(849)
lasso_caret<- train(dat[,1:2],dat[,3], method = "glmnet", lambda= 0,
                    tuneGrid = expand.grid(alpha = 1,  lambda = 0))

This would provide matching results that also match the output of simple glmnet.

lasso_raw <- glmnet( x= dat[,1:2], y = dat[,3], alpha = 1, lambda = 0)
ridge_raw <- glmnet( x= dat[,1:2], y = dat[,3], alpha = 0, lambda = 0)

all.equal(lasso_raw$beta, ridge_raw$beta) 
# TRUE
all.equal(ridge_raw$beta, ridge_caret$finalModel$beta)
# TRUE
all.equal(ridge_caret$finalModel$beta, lasso_caret$finalModel$beta)
# TRUE

As mentioned this seems to be unintentionally so you might wish to raise it as an issue in caret's github repo.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Side-note: If we want this to match the estimates from lm we need to note that glmnet sets standardize = TRUE by default. This needs to be set to FALSE (alongside lambda =0 obviously) to match the output of lm. $\endgroup$ – usεr11852 Aug 22 '18 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.