# About Akaike's criterion and VC-dimension of linear regressors

Akaike's model selection criterion is usually justified on the base that the empirical risk of a ML estimator is a biased estimator of the true risk of the best estimator in the parametric family, say the family of linear regressors on a m-dimensional variable, $S_m$

On the other hand, this family, $S_m$, is known to have finite VC dimension ($VC = m+1$). Having finite VC-dimension should grant that the empirical risk minimizer is asymtotically consistent (Vapnik, "An overview of statistical learning theory")

What am I missing?

Thanks Jake

-

Although indeed the empirical risk of a ML estimator is biased, this does not contradict the asymptotic consistency because the bias depends on the sample size. Therefore, the empirical risk is asymptotically unbiased. However, since in practice we deal with finite sample sets, this does not mean we can always ignore the bias.

-

It's not clear what you're asking. The justification you list for AIC may well still be a valid one, just not something that applies to all parameteric families. You highlighted a particular family, linear regressors on an m-dimensional variable, where empirical risk minimization might be just fine. That doesn't speak to generic properties about different parameteric families, nor infinite VC dimension families.

From my perspective, I'm not sure I agree that empirical risk plays much of a role in the justification of AIC. As I understand it, AIC represents relative information loss between two competing models, and isn't necessarily intended to be thought of in terms of the optimal choice from a family of parametric models. If you're choosing between a few different models, you might like to minimize the asymptotic KL-Divergence from the "true" model, and it is for this that AIC serves as a proxy.

As an aside, I personally advocate ignoring appeals to unbiasedness in estimators. It's not a genuinely meaningful way to evaluate estimators. Andrew Gelman recently had a short post about this.

-