# What is AIC? Looking for a formal but intuitive answer

I've heard that AIC can be used to choose among several models (which regressor to use).

But i would like to understand formally what it is in a kind of "advanced undergraduated" level, which I think would be something formal but with intuition arising from the formula.

And is it possible to implement AIC in stata with complex survey data?

• I would suggest reviewing the wiki link: en.wikipedia.org/wiki/Akaike_information_criterion and then edit the question so that you can highlight the aspect of AIC that you do not understand. – user28 Jul 27 '10 at 18:14
• Read also this question: stats.stackexchange.com/questions/577/… – user88 Jul 27 '10 at 18:22
• Consider asking the general question about the AIC separately from the stata question. – russellpierce Jul 28 '10 at 0:16

Let $f$ be your true distribution, and $g$ the family from which you are trying to fit your data. Then $\theta$, the maximum likelihood estimator of parameters of $g$, is a random variable. You could formulate model selection as finding the distribution family $g$ that minimizes the expected KL divergence between $f$ and $g(\theta)$, which can be written as

$$\text{Entropy}(f)-E_x E_y[\log(g(x|\theta(y)))]$$

Since you are minimizing over $g$, the Entropy($f$) term doesn't matter and you look for $g$ that maximizes $E_x E_y[\log(g(x|\theta(y)))]$.

Let $L(\theta(y)|y)$ be the likelihood of data $y$ according to $g(\theta)$. You could estimate $E_x E_y[\log(g(x|\theta(y)))]$ as $\log(L(\theta(y)|y))$ but that estimator is biased.

Akaike's showed that when $f$ belongs to family $g$ with dimension $k$, the following estimator is asymptotically unbiased

$$\log(L(\theta(y)|y))-k$$

Burnham has more details in this paper, also blog post by Enes Makalic has further explanation and references

It is a heuristic, and as such, has been subjected to extensive testing. So when to trust it or not is not simple clear-cut and always-true decision.

At a rough approximation, it trades off goodness of fit and number of variables ("degrees of freedom"). Much more, as usual, at the Wikipedia article about AIC.

Basically one needs a loss function in order to optimize anything. AIC provides the loss function which when minimized gives a "optimal"* model which fits the given data. The AIC loss function (2k-2*log(L)) tries to formulate the bias variance trade off that every statistical modeler faces when fitting a model to finite set of data.

In other words while fitting a model if you increase the number of parameters you will improve the log likelihood but will run into the danger of over fitting. The AIC penalizes for increasing the number of parameters thus minimizing the AIC selects the model where the improvement in log likelihood is not worth the penalty for increasing the number of parameters.

• Note that when I say optimal model it is optimal in the sense that the model minimizes the AIC. There are other criteria (e.g. BIC) which may give other "optimal" models.

I don't have any experience with stata so cannot help you with the other part of the question.