1
$\begingroup$

I am doing a comparison between two parametrical non-linear models with different amount of parameters. I am using Akaike information criterion (AIC) with the following formula:

2*k + N * log(RSS / N)

where k - number of parameters, N - number of experimental points and RSS - sum of squared errors

Models describe physical property of the system which consists of 19 structural units and each of them have 3 (model 1) or 4 (model 2) adjustable bounded parameters.

For model 1 I have :

RSS = 17283, N = 5128, k = 19*3 = 57, so AIC = 2 * 57 + 5128 * log(17283/5128) = 2819.9

For model 2:

RSS = 10115, N = 5128, k = 19*4 = 76, so AIC = 2 * 76 + 5128 * log(10115/5128) = 1664.9

If my calculation is correct, i should choose model 2, but what bothers me is that the amount of parameters here doesn't play a significant role, I could increase it up to 500 and the second model would still be better, which seems counter-intuitive. Is this calculation correct, or am I missing something important?

| cite | improve this question | |
$\endgroup$
  • 2
    $\begingroup$ Look at that huge percentage drop (>40%!) in RSS between the two models. With 5128 observations, that isn't due to chance. Those extra 19 parameters are giving you a lot of benefit! $\endgroup$ – jbowman Mar 12 '18 at 15:58
  • $\begingroup$ @jbowman, it still feels like the Ocaam's razor of AIC should be stricter here. Thanks for the answer! $\endgroup$ – K. Starodub Mar 13 '18 at 8:10
  • $\begingroup$ Your calculations aren't correct, BTW, the first AIC should be 6344.6 and the second 3635.5. You were using log - base 10, not log - base $e$, hence the difference. $\endgroup$ – jbowman Mar 15 '18 at 15:37
  • $\begingroup$ ok, I see, the use of log instead of ln for the base e confused me here, thanks $\endgroup$ – K. Starodub Mar 16 '18 at 11:38
1
$\begingroup$

Let's examine this in an empirical way, with the objective of building intuition about the effects of sample size and differences between RSSs. We'll simulate many samples of size $N=(5128-57) = 5085$ from a Normal distribution with mean $0$ and variance $\sigma_1^2 = 17283/5085 = 3.399$, calculate $2k + N\log(\text{RSS}/N)$, plot a histogram. The numbers have been chosen to match those from the model 1 results. We'll repeat the exercise with numbers chosen to match the model 2 results, and, by comparing where the two histograms lie, we'll be able to see whether the two AICs really have any chance of being different due to randomness or not.

N <- c(5085,5066)
k <- c(57,76)
s2 <- c(3.399, 1.997)

aic_res <- rep(0,20000)

for (j in 1:2) {
  shift <- (j-1)*10000
  for (i in 1:10000) {
    x <- rnorm(N[j], 0, sqrt(s2[j]))
    aic_res[i+shift] <- 2*k[j] + N[j]*log(sum(x*x)/N[j])
  }
}

hist(aic_res, breaks=25,
     main="Histogram of sample AICs",
     xlab="Model 1,2 AIC")

leading to:

enter image description here

We can see there is no chance (in the practical sense of the word) that the difference between these two AICs is due to chance (in the statistical sense of the word.) The gap between the two AICs is several times the width of the histograms themselves.

How big a $k$ would be needed for these two histograms to more-or-less align? An easy calculation: $(6344.6-3635.5)/2 = 1354$.

So, what have you missed? The large sample size combined with the large difference between the two RSSs is such that a mere 19 extra parameters is, by itself, a small factor in the overall AIC calculation. There are two ways to look at this: how small a sample size would we need in order for the two histograms to more-or-less overlap, and how small a difference in the observed RSS values would we need in order for the two histograms to more-or-less overlap given the original sample size.

In the first case, reducing the sample sizes to $(169, 150)$ gives us the following plot:

enter image description here

which still would favor model 2, but not by much.

In the second case, a quick series of calculations shows us that the two AICs will be roughly the same if the RSS from model 2 is $17155.4$, an 0.74% decrease from the model 1 RSS of $17283$. Given that the actual decrease is far greater than that, it's clear that Model 2 is a better one.

These experiments were performed with Normal distributions, and other distributions would give you different results - for example using a $t(3)$ distribution would cause the histograms to spread out a little - but, given the magnitude of the differences between RSS and the large sample size, the qualitative results wouldn't change.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.