0
$\begingroup$

I am concerned with the formulas I've seen for the AIC or BIC when using a squared-errors instead of likelihood.

On the AIC wikipedia page, there is the cryptic formula for LL:

$ -{\frac {n}{2}}\ln(2\pi )-{\frac {n}{2}}\ln(\sigma ^{2})-{\frac {1}{2\sigma ^{2}}}\mathrm {RSS} =-{\frac {n}{2}}\ln(\mathrm {RSS} /n)+C_{1} $

Similar formulas can be found in many places on the internet [2].

I understand how the left hand side comes about, from the $\log P$ of the density of a normal distribution. But

  • how about the right-hand side? Where does the logarithm come from?
  • Moreover, I am worried that using $\ln(RSS/n)$ depends on the scale / units of the variable that I'm using. For example, if the error term is measured in mm rather than km, then surely the AIC values would be vastly different?
  • If the AIC varies with the units, how could it be interpreted -- e.g. with a criterion of $\Delta AIC > 2$? It might make sense if I divided $RSS$ by the total data variance or something?
  • Furthermore, someone told me never to take a $\log$ of a quantity with units - so how can this be correct?

--

Edit

I have found this paper which shows how the RHS is derived. Apparently, the term $RSS/2\sigma^2$ simply cancels out because the estimated $\sigma^2$ (in a least squares model) simply is the $RSS$. So the $\ln(RSS/n)$ term just comes from the scaling factor of the normal distribution (the $1/\sigma$ coefficient).

So I am now even more puzzled about how to interpret the AIC, since it clearly changes when I change the scale of my measurement. E.g. if I measure in millimeters, the model's likelihood is much lower.

$\endgroup$
0
$\begingroup$

OK think I have solved it.

I found this closely related but unanswered question, but the comments helped me work it out.

If I measure in different units, then all the $RSS$ and $\sigma$get multiplied by a constant. Thus $log(RSS/n)$ is the same but for an additive constant.

As long as $AIC$s being compared have data in the same units, it is a valid comparison. This is because $AIC_1 - AIC_2$ is effectively testing $\frac{RSS_1}{RSS_2}$ i.e. it is similar to using a ratio of error variances, which seems good -- e.g. just as you might do in an $F$ test.

In fact, I wonder if all information-theoretic measures applied to continuous domains are only valid "up to an additive constant". This is because they are just generalisations of a discrete case, that depend on the "granularity". Similar arguments seem to apply to entropy measures.

This is all OK as long as the model is linear and accordingly the space being modelled is truly linear. This works if the scale also has an additive constant (such as Fahrenheit vs Centigrade) as long as the model contains an intercept, because only residuals are used.

I found a short article that illustrates a few cautions of this method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.