Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I've heard that AIC can be used to choose among several models (which regressor to use).
But i would like to understand formally what it is in a kind of "advanced undergraduated" level, which I think would be something formal but with intuition arising from the formula.
And is it possible to implement AIC in stata with complex survey data?
Let $f$ be your true distribution, and $g$ the family from which you are trying to fit your data. Then $\theta$, the maximum likelihood estimator of parameters of $g$, is a random variable. You could formulate model selection as finding the distribution family $g$ that minimizes the expected KL divergence between $f$ and $g(\theta)$, which can be written as
$$\text{Entropy}(f)-E_x E_y[\log(g(x|\theta(y)))]$$
Since you are minimizing over $g$, the Entropy($f$) term doesn't matter and you look for $g$ that maximizes $E_x E_y[\log(g(x|\theta(y)))]$.
Let $L(\theta(y)|y)$ be the likelihood of data $y$ according to $g(\theta)$. You could estimate $E_x E_y[\log(g(x|\theta(y)))]$ as $\log(L(\theta(y)|y))$ but that estimator is biased.
Akaike's showed that when $f$ belongs to family $g$ with dimension $k$, the following estimator is asymptotically unbiased
$$\log(L(\theta(y)|y))-k$$
Burnham has more details in this paper, also blog post by Enes Makalic has further explanation and references
Basically one needs a loss function in order to optimize anything. AIC provides the loss function which when minimized gives a "optimal"* model which fits the given data. The AIC loss function (2k-2*log(L)) tries to formulate the bias variance trade off that every statistical modeler faces when fitting a model to finite set of data.
In other words while fitting a model if you increase the number of parameters you will improve the log likelihood but will run into the danger of over fitting. The AIC penalizes for increasing the number of parameters thus minimizing the AIC selects the model where the improvement in log likelihood is not worth the penalty for increasing the number of parameters.
I don't have any experience with stata so cannot help you with the other part of the question.
It is a heuristic, and as such, has been subjected to extensive testing. So when to trust it or not is not simple clear-cut and always-true decision.
At a rough approximation, it trades off goodness of fit and number of variables ("degrees of freedom"). Much more, as usual, at the Wikipedia article about AIC.
