# How can I apply Akaike Information Criterion and calculate it for Linear Regression?

I tried to develop a Linear Regression model and want to understand how to measure models with a different combination of variables with Akaike's Information Criterion.

Consider the Linear Regression below:

$ChildenScore = \alpha + \beta_1FatherEducation + \beta_2MotherEducation + \beta_3FatherIncome + \beta_4MotherIncome$

I'd like to try AIC to choose the best model. Hope to hear some explanations.

A simple formula for the calculation of the AIC in the OLS framework (since you say linear regression) can be found in Gordon (2015, p. 201):

$$\text{AIC} = n *\ln\Big(\frac{SSE}{n}\Big)+2k$$

Where SSE means Sum of Squared Errors ($\sum(Y_i-\hat Y_i)^2$), $n$ is the sample size, and $k$ is the number of predictors in the model plus one for the intercept. Although AIC values are not generally interpretable, differences between values for different models can be interpreted (A number of questions on CV covers this issue, for example in here). So, the model with the smallest AIC is usually selected. It is easy to see why this is the case in the above formula: All else being equal, as the SSE decreases, AIC also decreases.

In other sources, you may find a more general, maximum-likelihood formula. For example, in Applied Regression Analysis and Generalized Linear Models, Fox provides:

$$\text{AIC}_j \equiv - \text{log}_eL(\hat \theta_j)+2s_j$$

where $L(\hat \theta_j)$ is the maximized likelihood under model $M_j$ ($\theta_j$ is the vector of parameters of the model, $\hat \theta_j$ is the vector of maximum-likelihood estimates of the parameters) and $s_j$ is the number of parameters (Fox 2016, p. 673-674). In OLS framework, this second formula simplifies to the first one. Using this first formula, I think it is not difficult to calculate and use AIC for the comparison of linear regression models.

Fox, J. (2016). Applied Regression Analysis and Generalized Linear Models (3rd ed.). Los Angeles: Sage Publications.

Gordon, R. A. (2015). Regression Analysis for the Social Sciences. New York and London: Routledge.

And the original article:

Akaike, H. (1998). Information Theory and an Extension of the Maximum Likelihood Principle. In E. Parzen, K. Tanabe, & G. Kitagawa (Eds.), Selected Papers of Hirotogu Akaike (pp. 199–215). New York: Springer.

• Thank you for the explanation and it helps me step further into it. Feb 12, 2017 at 8:53
• Can you use the same formula from Gordon (2015, p. 201) with a tree regressor model or is that invalid? Aug 14, 2020 at 17:26
• @PyRsquared sorry for late reply. Honestly, I don’t know. I am not experienced with tree regressor models, but I’ll let you know if I can find something useful. Aug 22, 2020 at 12:07
• Looking at en.wikipedia.org/wiki/Akaike_information_criterion I find the well known $\ln\mathcal{L}(\mu,\sigma) \, = \, -\frac{n}{2}\ln(2\pi) - \frac{n}{2}\ln\sigma^2 - \frac{1}{2\sigma^2}\sum_{i=1}^n (x_i-\mu)^2$ for assuming that the errors scatter with a normal distribution around the model. If AIC =\ln\mathcal{L} +2k why do you and e.g. statsmodels github.com/statsmodels/statsmodels/blob/… compute AIC using log likelihood function llf=-log(SSR)+... shouldn't it be llf=-SSR+...(no log) Jul 28, 2022 at 8:26
• I see you have the answer here: stats.stackexchange.com/questions/583479/… . So, I removed my comment. Just a note: wiki page you linked gives an idea about this under "Comparison with least squares." Jul 28, 2022 at 12:20