38
$\begingroup$

Is it possible to calculate AIC or BIC values for lasso regression models and other regularized models where parameters are only partially entering the equation. How does one determine the degrees of freedom?

I'm using R to fit lasso regression models with the glmnet() function from the glmnet package, and I'd like to know how to calculate AIC and BIC values for a model. In this way I might compare the values with models fit without regularization. Is this possible to do?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Yes you can do this, but it will most likely require you to derive the proper correction. The correction is derived in this paper ncbi.nlm.nih.gov/pmc/articles/PMC2629611 in the context of penalized finite mixture modeling, but an analogous argument would suffice in other penalized models. $\endgroup$
    – Macro
    Commented Apr 4, 2012 at 5:58

3 Answers 3

Reset to default
13
$\begingroup$

You may also find the following papers to be of interest:

R. J. Tibshirani and J. Taylor (2011), Degrees of freedom in lasso problems, arXiv preprint:1111.0653.

H. Zou, T. Hastie and R. Tibshirani (2007), On the degrees of freedom of the lasso, Annals of Statistics 35 (5), 2173–2192.

Improve this answer
$\endgroup$
1
  • 20
    $\begingroup$ would have been nice if there was a bit of explanation on the final answer; $\endgroup$
    – Areza
    Commented Nov 17, 2016 at 1:41
18
$\begingroup$

I was struggling a lot with a way how to calculate AIC and BIC for glmnet models. However, after quite a lot of searching, I found on the third page of google results the answer. It can be found here. I am posting it here for future readers as I believe I cannot be the only one.

In the end, I implemented the AIC and BIC in the following way:

fit <- glmnet(x, y, family = "multinomial") 

tLL <- fit$nulldev - deviance(fit)
k <- fit$df
n <- fit$nobs
AICc <- -tLL+2*k+2*k*(k+1)/(n-k-1)
AICc

BIC<-log(n)*k - tLL
BIC
Improve this answer
$\endgroup$
2
$\begingroup$

In the link referenced by johnnyheineken, the author states:

I'm afraid that the two quantities available from the glmnet object(dev.ratio, nulldev) are not enough to obtain the likelihood for the model, which you need to compute AICc. You have two equations in three unknowns: likelihood(null), likelihood(model), and likelihood(saturated). I can't get the likelihood(model) free from the likelihood(null).

It seems to me, that if you're comparing the AIC between two models, the fact that you can't separate the null deviance shouldn't matter. Since it exists on both "sides" of the inequality, it would show which model must have the lower AIC. This is dependent on two things:

  1. The data is the same in both models (necessary for AIC comparison anyway)
  2. I am neither forgetting something from Stat101 nor high school algebra (a strong assumption given my current caffeine intake)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.