25
$\begingroup$

When computing AIC,

$AIC = 2k - 2 ln L$

k means 'number of parameters'. But what counts as a parameter? So for example in the model

$y = ax + b$

Are a and b always counted as parameters? What if I don't care about the value of the intercept, can I ignore it or does it still count?

What if

$y = a f(c,x) + b$

where $f$ is a function of c and x, do I now count 3 parameters?

Improve this question
$\endgroup$
4
  • 10
    $\begingroup$ This is a good question, because there is a subtlety: $k$ is the number of identifiable parameters to be estimated. For instance, although in the regression model $Y\sim\mathcal{N}(\beta_0+\beta_1X_1+\beta_2X_2+\beta_3(X_1+X_2), \sigma^2)$ five parameters are written, nevertheless $k=4$. (This model is equivalent to $Y\sim\mathcal{N}(\beta_0+\alpha_1X_1+\alpha_2X_2, \sigma^2)$ with $\alpha_1=\beta_1+\beta_3$ and $\alpha_2=\beta_2+\beta_3$, which explicitly needs only four parameters.) $\endgroup$
    – whuber ♦
    Jun 16, 2014 at 14:09
  • 3
    $\begingroup$ Strictly, you count all identifiable, free parameters - mean parameters, shape and scale parameters, whatever (and it matters for AIC$_C$), but for AIC it's of no consequence if you omit parameters common to the models being compared. So for example, in regression, you should count the variance parameter. Hence, by my count, all your parameter counts in your question are one short - but if there's exactly one in all the models, it doesn't hurt to drop it for AIC. R explicitly counts the variance parameter when computing AIC in regression models. $\endgroup$
    – Glen_b
    Jun 16, 2014 at 14:27
  • $\begingroup$ @whuber Why is this excellent comment not posted as an answer? :) $\endgroup$
    – Alexis
    Jun 16, 2014 at 15:17
  • $\begingroup$ Thank you, @Alexis. I posted this thought as a comment because the idea is adequately covered in P Schnell's answer: I wished only to emphasize it a little more. $\endgroup$
    – whuber ♦
    Jun 16, 2014 at 15:51

3 Answers 3

Reset to default
19
$\begingroup$

As mugen mentioned, $k$ represents the number of parameters estimated. In other words, it's the number of additional quantities you need to know in order to fully specify the model. In the simple linear regression model $$y=ax+b$$ you can estimate $a$, $b$, or both. Whichever quantities you don't estimate you must fix. There is no "ignoring" a parameter in the sense that you don't know it and don't care about it. The most common model that doesn't estimate both $a$ and $b$ is the no-intercept model, where we fix $b=0$. This will have 1 parameter. You could just as easily fix $a=2$ or $b=1$ if you have some reason to believe that it reflects reality. (Fine point: $\sigma$ is also a parameter in a simple linear regression, but since it's there in every model you can drop it without affecting comparisons of AIC.)

If your model is $$y=af(c,x)+b$$ the number of parameters depends on whether you fix any of these values, and on the form of $f$. For example, if we want to estimate $a, b, c$ and know that $f(c,x)=x^c$, then when we write out the model we have $$y=ax^c+b$$ with three unknown parameters. If, however, $f(c,x)=cx$, then we have the model $$y=acx+b$$ which really only has two parameters: $ac$ and $b$.

It is crucial that $f(c,x)$ is a family of functions indexed by $c$. If all you know is that $f(c,x)$ is continuous and it depends on $c$ and $x$, then you're out of luck because there are uncountably many continuous functions.

Improve this answer
$\endgroup$
4
  • 2
    $\begingroup$ (+1) Perhaps worth mentioning that throughout "estimate" means "estimate by maximum-likelihood". $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jun 16, 2014 at 13:54
  • $\begingroup$ Does that really matter? In actual fact my $f(c,x)$ is an enormous simulation, impossible to pull apart analytically, and taking hours to compute. I try it with about 20 different values of $c$ because that's all we have time for, and I stick with the value of $c$ that gives the best $r^2$ at the end of the day. So in a manner of speaking I have estimated $c$ as best I can, though not as you would in a regression. Surely it still counts as a parameter for AIC though? $\endgroup$
    – Sideshow Bob
    Jun 18, 2014 at 11:49
  • 2
    $\begingroup$ @SideshowBob: Yes - when you compare two models the difference in maximized log likelihoods is a biased estimator of the difference in expected Kullback-Leibler information loss & the penalty term in AIC approximately corrects that bias. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jun 19, 2014 at 21:25
  • 1
    $\begingroup$ @SideshowBob: I should mention there are modifications of AIC for generalized estimating equations & the like - they use maximized quasi-likelihood & a rather more complex penalty term. $\endgroup$
    – Scortchi - Reinstate Monica ♦
    Jun 20, 2014 at 14:57
6
$\begingroup$

For any statistical model, the AIC value is $\mathit{AIC} = 2k - 2\ln(L)$ where k is the number of parameters in the model, and L is the maximized value of the likelihood function for the model.

(see here)

As you may see, $k$ represents the number of parameters estimated in each model. If you model includes an intercept (that is, if you compute a point estimate, variance and confidence interval for the intercept) then it counts as a parameter. On the other hand, if you are computing a model without an intercept, it does not count.

Remember that AIC does not only summarise goodness of fit but it also considers the complexity of the model. That's why $k$ exists, to penalise models with more parameters.

I don't feel knowledgeable enough to answer your second question, I'll leave it for another member of the community.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Does that mean if I Box-Cox transform both x and y, then $\lambda$ from each of those transforms counts as a parameter too? $\endgroup$
    – Sideshow Bob
    Jun 16, 2014 at 13:49
  • 1
    $\begingroup$ Yes, certainly. $\endgroup$
    – PA6OTA
    Jun 16, 2014 at 18:38
1
$\begingroup$

First, to those who may not be familiar with AIC: the Akaike Information Criterion (AIC) is a simple metric designed to compare the "goodness" of models.

According to AIC, when trying to choose between two different models applying to the same input and response variables, i.e. models designed to solve the same problem, the model with the lower AIC is considered "better".

In the AIC formula, $k$ refers to the number of variables (input features, or columns) in the model. The more complex the model is (more variables needed to get the estimate or prediction), the higher the AIC is. This ensures that among two models with the same predictive power or accuracy, the simpler model wins. This is a form of Occam's razor.

So the simple answer to the last question is: if the c in $f(c, x)$ is a constant that doesn't change with the observations, then, it should not be included in $k$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.