7
$\begingroup$

suppose I have a data set $x_1, \ldots, x_n$ and I would fit a normal, an exponential and a uniform distribution to them. The fitting function spits out a bunch of goodness-of-fit statistics, e.g. the AIC, BIC, chi-square, Kolmogorov-Smirnov, etc.

I am trying to convince someone that the AIC is not appropriate here, because we have different log-likelihoods, and sometimes different number of parameters, depending on the distributions. I would prefer the p-value of the Kolmogorov-Smirnov-Test to compare the fits.

Is my approach justified? How can I convince my coworker the AIC is not okay here (he likes to see a cited paper or something equivalent)?

edit: Specifically, I was shown this article: http://www.vosesoftware.com/whitepapers/Fitting%20distributions%20to%20data.pdf

I have no idea what to say to this. Page 4 lists the flaws of the chi-squared, Kolmogorov-Smirnov etc, and page 5 and 6 praise the AIC. Is he right?

Improve this question
$\endgroup$
9
  • 4
    $\begingroup$ I think it is valid to use AIC for comparing these models. The Wikipedia entry mentions "There are, however, important distinctions. In particular, the likelihood-ratio test is valid only for nested models whereas AIC (and AICc) has no such restriction". On the other hand, it is more intriguing why are you comparing such diferent models ""I would fit a normal, an exponential and a uniform distribution"? They are different in terms of shape, support, ... $\endgroup$
    – user10525
    Commented Jul 6, 2012 at 12:48
  • $\begingroup$ I believe the author is a bit biased to information criteria, which are not the Panacea by the way because they are based on asymptotic results. I think it is better to use a couple of criteria that assess different features of the models. For example, AIC penalises the number of parameters, some goodness of fit tests assess the fit on the tails or the shoulders of the distribution, and some other evaluate the predictive performance of the models in question. $\endgroup$
    – user10525
    Commented Jul 6, 2012 at 13:03
  • $\begingroup$ I am comparing such different models just because we want to see all possible models. Our function just fits all models that are interesting to us - I know they are quite different in character :) In part, I also wanted to deliberately choose different models to see if even there the AIC could be used (if it could at all). $\endgroup$
    – Alexander Engelhardt
    Commented Jul 6, 2012 at 13:04
  • 1
    $\begingroup$ Why are you trying to convince them that AIC is not justified? $\endgroup$
    – John
    Commented Jul 6, 2012 at 13:44
  • 1
    $\begingroup$ @AlexxHardt Some related questions/answers: "Can AIC compare across different types of model?", "Non-nested model selection", "Testing the difference in AIC of two non-nested models". $\endgroup$
    – user10525
    Commented Jul 9, 2012 at 14:18

2 Answers 2

Reset to default
10
$\begingroup$

You have to penalize the model for number of parameters. Let's say you had 30 data points and a model to fit it to that takes 29 parameters to define it. You could fit the data perfectly. But, that's not a terribly fair way to compare it to a uniform distribution with far fewer parameters.

The paper you cite mentions this. Likely you're having trouble making an argument against it because there isn't a good general one. The argument would be against how much you penalize for extra parameters in the model. In that case you may want to examine different kinds of information criteria.

Furthermore, it's also a good idea to look at some other fit measures as well. There's nothing wrong with using multiple ones and making rational arguments when the AIC differences are very small.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Oh, yes, I agree with the need for penalty. What I don't agree with, is that the likelihood descend from different distributions, i.e. different density families. Aren't we comparing apples with oranges there? I thought the AIC only works for nested models where you add one input variable to your regression and then see how much the likelihood improves. But it would still be the same likelihood family. $\endgroup$
    – Alexander Engelhardt
    Commented Jul 9, 2012 at 11:21
  • 2
    $\begingroup$ @AlexxHardt This is not quite correct (as I have mentioned in my first comment). Please, have a look at this document about AIC Myths, particularly the last paragraph of pp. 2. Other references by Brian Ripley: 1, 2, 3. $\endgroup$
    – user10525
    Commented Jul 9, 2012 at 13:40
1
$\begingroup$

I'd go further and say it is probably the most widely accepted method for comparing distributions. But you should really use the corrected AIC, which has a piece added to it to adjust for small sample sizes. See Burnham and Anderson 2002, for example.

This site will take a set of numbers and do the comparisons for you, using the corrected AIC mentioned above. http://www.easydatascience.com/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.