5
$\begingroup$

Many moons ago, I asked how to differentiate between two very similar non-linear fits and which was better. Finally got that all straightened out after many headaches and three different software packages -- calculating AIC, AICc, and BIC and all give a consistent result.

Now I'm trying to apply a slight modification to the data and re-fit. The question is, since $\chi^2$ is fairly useless on very non-linear models, can I use AIC to tell me if the original data or new data are better fits to that model? As in, I use an $f(x)$ model, fit it with dataset1, and then I fit the SAME model to dataset2, and I calculate AIC/AICc/BIC of both fits. Is that a meaningful comparison, or can they only be used to rate between two different fits to the same data?

$\endgroup$
3
  • $\begingroup$ Why not simply compare $s^2$? $\endgroup$
    – Glen_b
    Commented Dec 13, 2012 at 22:12
  • $\begingroup$ Sorry, not a stats person ... $s^2$ is ... ? $\endgroup$ Commented Dec 13, 2012 at 22:30
  • $\begingroup$ The residual standard error. Essentially, the standard deviation of the residuals. It's a measure of how close the points are to your nonlinear model. It, or $s$, should be an output of your nonlinear fit. $\endgroup$
    – Glen_b
    Commented Dec 13, 2012 at 22:34

2 Answers 2

7
$\begingroup$

The comparison is utterly meaningless.

Consider, for instance, fitting iid data $\mathbb{x} = (x_1, \ldots, x_n)$ to a location-scale family having a continuous pdf

$$f(x; \mu, \sigma) = f\left(\frac{x-\mu}{\sigma}\right) / \sigma.$$

The log likelihood for these data equals

$$\Lambda(\mathbb{x}; \mu, \sigma) = \sum_{i=1}^n \log \left(f\left(\frac{x_i-\mu}{\sigma}\right)/\sigma\right) = -n \log(\sigma) + \sum_{i=1}^n \log f\left(\frac{x_i-\mu}{\sigma}\right).$$

Because $\frac{x-\mu}{\sigma} = \frac{\alpha x-\alpha\mu}{\alpha\sigma}$ for any positive real number $\alpha$, $(\hat{\mu}, \hat{\sigma})$ maximize the log likelihood for $\mathbb{x}$ if and only if $(\alpha \hat{\mu}, \alpha \hat{\sigma})$ maximize the log likelihood for a corresponding dataset $(\alpha x_1, \ldots, \alpha x_n)$ obtained by rescaling the original data. This is tantamount to a change in their units of measurement, so the "fit" to the model cannot be any better or worse. However,

$$\Lambda(\mathbb{x}; \hat{\mu}, \hat{\sigma}) = n\log(\alpha) + \Lambda(\alpha\mathbb{x}; \alpha\hat{\mu}, \alpha\hat{\sigma}).$$

It follows that we may change the optimal value of the log likelihood by any arbitrary amount $n\log(\alpha)$ by means of an appropriate choice of $\alpha$ without changing the fit (at least not in any statistically reasonable sense). For instance, if your data are lengths measured in meters and you were to re-express those lengths in kilometers, you would thereby increase the maximum log likelihood by $n\log(1000)$ without changing the "fit."

Because both AIC and BIC differ from the optimal value of $\Lambda$ by functions depending only on $n$, which does not change, the same conclusion holds for them, too.

$\endgroup$
1
$\begingroup$

Given the model has stayed the same, the number of parameters component of AIC is irrelevant. Than means the remaining component of the AIC is the maximised log likelihood.

So if the new AIC < old AIC then the log likelihood of the data under the model should have increased.

I'm not sure what this tells you in the case of having changed the data, but I assume you have your reasons.

$\endgroup$
1
  • $\begingroup$ In plain English, if the log likelihood of the data increased (as in AIC decreased), then that means ... ? FYI, the reason for slightly modifying the data is that I have what I measured, but some folks think that one should apply a dynamic correction based on location. I want to be able to say whether that makes the fit better or not. $\endgroup$ Commented Dec 13, 2012 at 22:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.