# Can AIC determine which data better fit the same model?

Many moons ago, I asked how to differentiate between two very similar non-linear fits and which was better. Finally got that all straightened out after many headaches and three different software packages -- calculating AIC, AICc, and BIC and all give a consistent result.

Now I'm trying to apply a slight modification to the data and re-fit. The question is, since $\chi^2$ is fairly useless on very non-linear models, can I use AIC to tell me if the original data or new data are better fits to that model? As in, I use an $f(x)$ model, fit it with dataset1, and then I fit the SAME model to dataset2, and I calculate AIC/AICc/BIC of both fits. Is that a meaningful comparison, or can they only be used to rate between two different fits to the same data?

-
Why not simply compare $s^2$? –  Glen_b Dec 13 '12 at 22:12
Sorry, not a stats person ... $s^2$ is ... ? –  Stuart Robbins Dec 13 '12 at 22:30
The residual standard error. Essentially, the standard deviation of the residuals. It's a measure of how close the points are to your nonlinear model. It, or $s$, should be an output of your nonlinear fit. –  Glen_b Dec 13 '12 at 22:34

The comparison is utterly meaningless.

Consider, for instance, fitting iid data $\mathbb{x} = (x_1, \ldots, x_n)$ to a location-scale family having a continuous pdf

$$f(x; \mu, \sigma) = f\left(\frac{x-\mu}{\sigma}\right) / \sigma.$$

The log likelihood for these data equals

$$\Lambda(\mathbb{x}; \mu, \sigma) = \sum_{i=1}^n \log \left(f\left(\frac{x_i-\mu}{\sigma}\right)/\sigma\right) = -n \log(\sigma) + \sum_{i=1}^n \log f\left(\frac{x_i-\mu}{\sigma}\right).$$

Because $\frac{x-\mu}{\sigma} = \frac{\alpha x-\alpha\mu}{\alpha\sigma}$ for any positive real number $\alpha$, $(\hat{\mu}, \hat{\sigma})$ maximize the log likelihood for $\mathbb{x}$ if and only if $(\alpha \hat{\mu}, \alpha \hat{\sigma})$ maximize the log likelihood for a corresponding dataset $(\alpha x_1, \ldots, \alpha x_n)$ obtained by rescaling the original data. This is tantamount to a change in their units of measurement, so the "fit" to the model cannot be any better or worse. However,

$$\Lambda(\mathbb{x}; \hat{\mu}, \hat{\sigma}) = n\log(\alpha) + \Lambda(\alpha\mathbb{x}; \alpha\hat{\mu}, \alpha\hat{\sigma}).$$

It follows that we may change the optimal value of the log likelihood by any arbitrary amount $n\log(\alpha)$ by means of an appropriate choice of $\alpha$ without changing the fit (at least not in any statistically reasonable sense). For instance, if your data are lengths measured in meters and you were to re-express those lengths in kilometers, you would thereby increase the maximum log likelihood by $n\log(1000)$ without changing the "fit."

Because both AIC and BIC differ from the optimal value of $\Lambda$ by functions depending only on $n$, which does not change, the same conclusion holds for them, too.

-