Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Many moons ago, I asked how to differentiate between two very similar non-linear fits and which was better. Finally got that all straightened out after many headaches and three different software packages -- calculating AIC, AICc, and BIC and all give a consistent result.

Now I'm trying to apply a slight modification to the data and re-fit. The question is, since $\chi^2$ is fairly useless on very non-linear models, can I use AIC to tell me if the original data or new data are better fits to that model? As in, I use an $f(x)$ model, fit it with dataset1, and then I fit the SAME model to dataset2, and I calculate AIC/AICc/BIC of both fits. Is that a meaningful comparison, or can they only be used to rate between two different fits to the same data?

share|improve this question
    
Why not simply compare $s^2$? –  Glen_b Dec 13 '12 at 22:12
    
Sorry, not a stats person ... $s^2$ is ... ? –  Stuart Robbins Dec 13 '12 at 22:30
    
The residual standard error. Essentially, the standard deviation of the residuals. It's a measure of how close the points are to your nonlinear model. It, or $s$, should be an output of your nonlinear fit. –  Glen_b Dec 13 '12 at 22:34

2 Answers 2

up vote 5 down vote accepted

The comparison is utterly meaningless.

Consider, for instance, fitting iid data $\mathbb{x} = (x_1, \ldots, x_n)$ to a location-scale family having a continuous pdf

$$f(x; \mu, \sigma) = f\left(\frac{x-\mu}{\sigma}\right) / \sigma.$$

The log likelihood for these data equals

$$\Lambda(\mathbb{x}; \mu, \sigma) = \sum_{i=1}^n \log \left(f\left(\frac{x_i-\mu}{\sigma}\right)/\sigma\right) = -n \log(\sigma) + \sum_{i=1}^n \log f\left(\frac{x_i-\mu}{\sigma}\right).$$

Because $\frac{x-\mu}{\sigma} = \frac{\alpha x-\alpha\mu}{\alpha\sigma}$ for any positive real number $\alpha$, $(\hat{\mu}, \hat{\sigma})$ maximize the log likelihood for $\mathbb{x}$ if and only if $(\alpha \hat{\mu}, \alpha \hat{\sigma})$ maximize the log likelihood for a corresponding dataset $(\alpha x_1, \ldots, \alpha x_n)$ obtained by rescaling the original data. This is tantamount to a change in their units of measurement, so the "fit" to the model cannot be any better or worse. However,

$$\Lambda(\mathbb{x}; \hat{\mu}, \hat{\sigma}) = n\log(\alpha) + \Lambda(\alpha\mathbb{x}; \alpha\hat{\mu}, \alpha\hat{\sigma}).$$

It follows that we may change the optimal value of the log likelihood by any arbitrary amount $n\log(\alpha)$ by means of an appropriate choice of $\alpha$ without changing the fit (at least not in any statistically reasonable sense). For instance, if your data are lengths measured in meters and you were to re-express those lengths in kilometers, you would thereby increase the maximum log likelihood by $n\log(1000)$ without changing the "fit."

Because both AIC and BIC differ from the optimal value of $\Lambda$ by functions depending only on $n$, which does not change, the same conclusion holds for them, too.

share|improve this answer

Given the model has stayed the same, the number of parameters component of AIC is irrelevant. Than means the remaining component of the AIC is the maximised log likelihood.

So if the new AIC < old AIC then the log likelihood of the data under the model should have increased.

I'm not sure what this tells you in the case of having changed the data, but I assume you have your reasons.

share|improve this answer
    
In plain English, if the log likelihood of the data increased (as in AIC decreased), then that means ... ? FYI, the reason for slightly modifying the data is that I have what I measured, but some folks think that one should apply a dynamic correction based on location. I want to be able to say whether that makes the fit better or not. –  Stuart Robbins Dec 13 '12 at 22:30

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.