AIC, BIC and GCV: what is best for making decision in penalized regression methods?

Question

My general understanding is AIC deals with the trade-off between the goodness of fit of the model and the complexity of the model.

$AIC =2k -2ln(L)$

$k$ = number of parameters in the model

$L$ = likelihood

Bayesian information criterion BIC is closely related with AIC.The AIC penalizes the number of parameters less strongly than does the BIC. I can see these two are used everywhere historically. But generalized cross validation (GCV) is new to me. How GCV can relate to BIC or AIC? How these criteria, together or separate used in selection of penalty term in panelized regression like ridge ?

Edit: Here is an example to think and discuss:

    require(lasso2)
    data(Prostate)
    require(rms)

    ridgefits = ols(lpsa~lcavol+lweight+age+lbph+svi+lcp+gleason+pgg45,
           method="qr", data=Prostate,se.fit = TRUE, x=TRUE, y=TRUE)
    p <- pentrace(ridgefits, seq(0,1,by=.01))
    effective.df(ridgefits,p)
    out <- p$results.all
    par(mfrow=c(3,2))
    plot(out$df, out$aic, col = "blue", type = "l", ylab = "AIC", xlab = "df"  )
    plot(out$df, out$bic, col = "green4", type = "l", ylab = "BIC",  xlab = "df" )
    plot(out$penalty, out$df,  type = "l", col = "red", 
     xlab = expression(paste(lambda)), ylab = "df" )
    plot(out$penalty, out$aic, col = "blue", type = "l",  
      ylab = "AIC", xlab = expression(paste(lambda))  )
    plot(out$penalty, out$bic, col = "green4", type = "l", ylab = "BIC", 
      xlab= expression(paste(lambda))

require(glmnet)
y <- matrix(Prostate$lpsa, ncol = 1)
x <- as.matrix (Prostate[,- length(Prostate)])
cv <- cv.glmnet(x,y,alpha=1,nfolds=10)
plot(cv$lambda, cv$cvm, col = "red", type = "l", 
      ylab = "CVM",   xlab= expression(paste(lambda))

Answer 1

I think of BIC as being preferred when there is a "true" low-dimensional model, which I think is never the case in empirical work. AIC is more in line with assuming that the more data we acquire the more complex a model can be. AIC using the effective degrees of freedom, in my experience, is a very good way to select the penalty parameter $\lambda$ because it is likely to optimize model performance in a new, independent, sample.

Answer 2

My own thoughts on this aren't very collected, but here is a collection of points I'm aware of that might help.

---

The Bayesian interpretation of AIC is that it is a bias-corrected approximation to the expected expected log pointwise predictive density, i.e. the out-of-sample prediction error. This interpretation is laid out nicely in Gelman, Hwang, and Vehtari (2013) and also discussed briefly on Gelman's blog. Cross-validation is a different approximation to the same thing.

Meanwhile, BIC is an approximation to the "Bayes factor" under a particular prior (explained nicely in Raftery, 1999). This is almost the Bayesian analogue of a likelihood ratio.

What's interesting about AIC and BIC is that penalized regression also has a Bayesian interpretation, e.g. LASSO is the MAP estimate of Bayesian regression with independent Laplace priors on the coefficients. A bit more info in this previous question and a lot more in Kyung, Gill, Ghosh, and Casella (2010).

This suggests to me that you might get some mileage, or at least a more coherent research design, by thinking and modeling in Bayesian terms. I know this is a bit unusual in a lot of applications like high-dimensional machine learning, and also somewhat removed from the (in my opinion) more interpretable geometric and loss-function interpretations of regularization. At the very least, I rely heavily on the Bayesian interpretation to decide between AIC and BIC and to explain the difference to laymen, non-statistically-oriented co-workers/bosses, etc.

I know this doesn't speak much to cross-validation. One nice thing about Bayesian inference is that it produces approximate distributions of your parameters, rather than point estimates. This, I feel, can be used to sidestep the issue of measuring one's uncertainty about prediction error. However, if you're talking about using CV to estimate hyperparameters, e.g. $\lambda$ for LASSO, I again defer to Gelman:

selecting a tuning parameter by cross-validation is just a particular implementation of hierarchical Bayes.