# AIC, BIC and GCV: what is best for making decision in penalized regression methods?

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model.

$AIC =2k -2ln(L)$

$k$ = number of parameters in the model

$L$ = likelihood

Bayesian information criterion BIC is closely related with AIC.The AIC penalizes the number of parameters less strongly than does the BIC. I can see these two are used everywhere historically. But generalized cross validation (GCV) is new to me. How GCV can relate to BIC or AIC? How these criteria, together or separate used in selection of penalty term in panelized regression like ridge ?

Edit: Here is an example to think and discuss:

    require(lasso2)
data(Prostate)
require(rms)

ridgefits = ols(lpsa~lcavol+lweight+age+lbph+svi+lcp+gleason+pgg45,
method="qr", data=Prostate,se.fit = TRUE, x=TRUE, y=TRUE)
p <- pentrace(ridgefits, seq(0,1,by=.01))
effective.df(ridgefits,p)
out <- p$results.all par(mfrow=c(3,2)) plot(out$df, out$aic, col = "blue", type = "l", ylab = "AIC", xlab = "df" ) plot(out$df, out$bic, col = "green4", type = "l", ylab = "BIC", xlab = "df" ) plot(out$penalty, out$df, type = "l", col = "red", xlab = expression(paste(lambda)), ylab = "df" ) plot(out$penalty, out$aic, col = "blue", type = "l", ylab = "AIC", xlab = expression(paste(lambda)) ) plot(out$penalty, out$bic, col = "green4", type = "l", ylab = "BIC", xlab= expression(paste(lambda)) require(glmnet) y <- matrix(Prostate$lpsa, ncol = 1)
x <- as.matrix (Prostate[,- length(Prostate)])
cv <- cv.glmnet(x,y,alpha=1,nfolds=10)
plot(cv$lambda, cv$cvm, col = "red", type = "l",
ylab = "CVM",   xlab= expression(paste(lambda))


I think of BIC as being preferred when there is a "true" low-dimensional model, which I think is never the case in empirical work. AIC is more in line with assuming that the more data we acquire the more complex a model can be. AIC using the effective degrees of freedom, in my experience, is a very good way to select the penalty parameter $\lambda$ because it is likely to optimize model performance in a new, independent, sample.

• Great practical interpretation, and also makes sense in the Bayesian context... "theoretic"-based likelihood ratio vs. "atheoretic" prediction error. Commented Jul 20, 2014 at 16:26
• It would probably help to elaborate on how "effective degrees of freedom" for a regularized solution can be computed and used in AIC. Commented Jul 20, 2014 at 16:33
• See the code in the R rms package effective.df function and my book Regression Modeling Strategies. The main idea, from Robert Gray, is that you consider the covariance matrix without penalization vs. the covariance matrix with penalization. The sum of the diagonal of a kind of ratio of these two gives you the effective d.f. Commented Jul 20, 2014 at 19:44
• @FrankHarrell: So if I understand you correctly - it is ok to compute a bunch of models in glmnet (each with a different lambda parameter) and compute the AIC for each model, and then choose the lambda corresponding to the model with the lowest AIC? This is basically another way for choosing the lambda parameter, other than using Cross Validation. Am I right? Commented Jan 16, 2018 at 20:52
• I was writing in the context of the rms package where a couple of the fitting functions when used with effective.df compute the effective number of parameters so you can get an effective AIC. This will approximate what you get from cross-validation with CV'ing. See this Commented Jan 16, 2018 at 22:24

My own thoughts on this aren't very collected, but here is a collection of points I'm aware of that might help.

The Bayesian interpretation of AIC is that it is a bias-corrected approximation to the expected log pointwise predictive density, i.e. the out-of-sample prediction error. This interpretation is laid out nicely in Gelman, Hwang, and Vehtari (2013) and also discussed briefly on Gelman's blog. Cross-validation is a different approximation to the same thing.

Meanwhile, BIC is an approximation to the "Bayes factor" under a particular prior (explained nicely in Raftery, 1999). This is almost the Bayesian analogue of a likelihood ratio.

What's interesting about AIC and BIC is that penalized regression also has a Bayesian interpretation, e.g. LASSO is the MAP estimate of Bayesian regression with independent Laplace priors on the coefficients. A bit more info in this previous question and a lot more in Kyung, Gill, Ghosh, and Casella (2010).

This suggests to me that you might get some mileage, or at least a more coherent research design, by thinking and modeling in Bayesian terms. I know this is a bit unusual in a lot of applications like high-dimensional machine learning, and also somewhat removed from the (in my opinion) more interpretable geometric and loss-function interpretations of regularization. At the very least, I rely heavily on the Bayesian interpretation to decide between AIC and BIC and to explain the difference to laymen, non-statistically-oriented co-workers/bosses, etc.

I know this doesn't speak much to cross-validation. One nice thing about Bayesian inference is that it produces approximate distributions of your parameters, rather than point estimates. This, I feel, can be used to sidestep the issue of measuring one's uncertainty about prediction error. However, if you're talking about using CV to estimate hyperparameters, e.g. $$\lambda$$ for LASSO, I again defer to Gelman:

selecting a tuning parameter by cross-validation is just a particular implementation of hierarchical Bayes.