2
$\begingroup$

Recently I was attempting to calculate the Akaike Information Criterion (AIC) for a set of fitted models in R, using standard packages available on CRAN. A package that I was using seemed to be exhibiting unexpected behavior, adding "+1" to the number of free parameters in each model. While trying to understand the reason for this result, I ran across this stackoverflow question about the same issue. I wrote my own question seeking further clarification, and confirmed that the extra parameter is intended to count the dispersion of the observed data about the fit model, even though the dispersion does not appear to be a parameter that could be independently adjusted in order to further improve the goodness of a fit. The result of this "+1" behavior is that if one performs a linear regression against, for example, a simple line, using maximum likelihood techniques to estimate a slope and intercept parameter, then logLik() will report the number of model parameters as 3, rather than 2. A respondent to my question pointed me to this additional answer further confirming that this is indeed the intended behavior, in fact a commit to the source code from 19 years ago even references switching back and forth between the two alternative return values, as if there had been some significant disagreement within the developer community about which option was the correct output.

I looked up Hirotugu Akaike's original article from 1974, and the abstract references the "number of parameters" in a model as meaning "the number of independently adjusted parameters within the model". Furthermore in the text of the paper (paywalled), in section IV, he additionally clarifies that "$k$ is the [...] number of parameters independently adjusted for the maximization of the likelihood" It seems clear from context that because the dispersion of a fitted model is not used or referenced in maximizing the likelihood, but only emerges after the fitting procedure is already completed, Akaike would argue that for purposes of calculating AIC, he did not intend for the dispersion to be counted as one of the model's free parameters. Thus, the practice of adding "+1" to the number of free parameters as the R stats module does, and using that value as input to an AIC calculation, seems counter to how the AIC was originally conceived by Akaike himself.

I am aware that in mathematics, sometimes we define certain concepts in order to enforce consistency within a larger theoretical framework. So, for example, we define that $\sqrt{-1} = i$, or $0! = 1$, even though the reasons for those definitions wouldn't make much sense if considered only in isolation, without an appreciation of the larger context. I'm curious whether there's also some broader context here that I'm not yet understanding, which developed within the field only after Akaike published his paper in 1974.

My question: what is the theoretical justification behind adding "+1" in order to count the dispersion among the number of free parameters in R's logLik() function, when the dispersion is not actually an independently adjustable parameter that could be tuned to further increase the maximum likelihood, and earlier thinkers such as even Akaike himself did not seem to have viewed it as such?

$\endgroup$

1 Answer 1

1
$\begingroup$

The loglikelihood returned for a linear regression model in R is the full Gaussian loglikelihood, which does depend on maximisation over $\sigma^2$; it isn't just the RSS.

The code (in stats:::logLik.lm) has

0.5 * (sum(log(w)) - N * (log(2 * pi) + 1 - log(N) + 
        log(sum(w * res^2))))

which is maximised over both $\beta$ and $\sigma^2$ and so genuinely has $|\beta|+1$ parameters. It is not $-\mathrm{RSS}/2$ or $-\mathrm{RSS}/2\sigma^2$

Because only differences in AIC have any meaning, the parameter count will only be affected if comparing a model with and a model without a dispersion parameter -- eg comparing a model with a prespecified error variance to one with an estimated error variance. In that setting, you would want to count $\sigma^2$ as a parameter.

$\endgroup$
4
  • $\begingroup$ AIC has an independent meaning of twice the negative expected loglikelihood on a new data point. It is thus an indirect measure of expected prediction loss where loss is defined w.r.t. the (log)likelihood. For that it is important to get the degrees of freedom right. $\endgroup$ Apr 30, 2021 at 7:00
  • 1
    $\begingroup$ Ok, but the Akaike paper was happy to drop additive constants to get a simpler form for the loglikelihood, so from that point of view he can't be regarded as definitive $\endgroup$ Apr 30, 2021 at 7:31
  • $\begingroup$ What I am trying to say is that only differences in AIC have any meaning is incorrect. (I am not trying to imply anything about what Akaike himself was thinking at some time point.) $\endgroup$ Apr 30, 2021 at 8:12
  • $\begingroup$ In the likelihood expression you cite, we identify the weights as $w = 1/\sigma^{2}$. In the question that I wrote for stackoverflow, I explicitly established that the default behavior for glm() at least (and probably lm() as well) does not actually adjust these weights to obtain a better fit. Therefore I think your first assertion is wrong: maximization (in the sense of a search over parameter space by an optimization algorithm such as Gauss-Newton or Levenberg-Marquardt) really is only occurring over $\beta$, not $\beta + \sigma^{2}$. $\endgroup$
    – stachyra
    Apr 30, 2021 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.