About Akaike's criterion and VC-dimension of linear regressors

Question

Akaike's model selection criterion is usually justified on the base that the empirical risk of a ML estimator is a biased estimator of the true risk of the best estimator in the parametric family, say the family of linear regressors on a m-dimensional variable, $S_m $

On the other hand, this family, $ S_m$, is known to have finite VC dimension ($VC = m+1$). Having finite VC-dimension should grant that the empirical risk minimizer is asymtotically consistent (Vapnik, "An overview of statistical learning theory")

What am I missing?

Thanks Jake

Answer 1

Although indeed the empirical risk of a ML estimator is biased, this does not contradict the asymptotic consistency because the bias depends on the sample size. Therefore, the empirical risk is asymptotically unbiased. However, since in practice we deal with finite sample sets, this does not mean we can always ignore the bias.

Answer 2

It's not clear what you're asking. The justification you list for AIC may well still be a valid one, just not something that applies to all parameteric families. You highlighted a particular family, linear regressors on an m-dimensional variable, where empirical risk minimization might be just fine. That doesn't speak to generic properties about different parameteric families, nor infinite VC dimension families.

From my perspective, I'm not sure I agree that empirical risk plays much of a role in the justification of AIC. As I understand it, AIC represents relative information loss between two competing models, and isn't necessarily intended to be thought of in terms of the optimal choice from a family of parametric models. If you're choosing between a few different models, you might like to minimize the asymptotic KL-Divergence from the "true" model, and it is for this that AIC serves as a proxy.

As an aside, I personally advocate ignoring appeals to unbiasedness in estimators. It's not a genuinely meaningful way to evaluate estimators. Andrew Gelman recently had a short post about this.

Answer 3

Burham and Anderson's maximized log-likelihood IS Vapkin's empirical risk estimate of the optimal linear regressor; their EyEx[log(g(x| ˆ θ(y)))] is Vapkin's generalization error.

While Burham and Anderson claim to have proven that the maximized log-likelihood (the empirical risk) is a biased estimate of the generalization error, such claim contrdicts a result by Vapkin that says that for classes with finite VC dimension the empirical risk is a consistent estimator of the generalization error. This is what I'm asking about!