# Is it okay to compare fitted distributions with the AIC?

suppose I have a data set $x_1, \ldots, x_n$ and I would fit a normal, an exponential and a uniform distribution to them. The fitting function spits out a bunch of goodness-of-fit statistics, e.g. the AIC, BIC, chi-square, Kolmogorov-Smirnov, etc.

I am trying to convince someone that the AIC is not appropriate here, because we have different log-likelihoods, and sometimes different number of parameters, depending on the distributions. I would prefer the p-value of the Kolmogorov-Smirnov-Test to compare the fits.

Is my approach justified? How can I convince my coworker the AIC is not okay here (he likes to see a cited paper or something equivalent)?

I have no idea what to say to this. Page 4 lists the flaws of the chi-squared, Kolmogorov-Smirnov etc, and page 5 and 6 praise the AIC. Is he right?

-
I think it is valid to use AIC for comparing these models. The Wikipedia entry mentions "There are, however, important distinctions. In particular, the likelihood-ratio test is valid only for nested models whereas AIC (and AICc) has no such restriction". On the other hand, it is more intriguing why are you comparing such diferent models ""I would fit a normal, an exponential and a uniform distribution"? They are different in terms of shape, support, ... –  user10525 Jul 6 '12 at 12:48
I believe the author is a bit biased to information criteria, which are not the Panacea by the way because they are based on asymptotic results. I think it is better to use a couple of criteria that assess different features of the models. For example, AIC penalises the number of parameters, some goodness of fit tests assess the fit on the tails or the shoulders of the distribution, and some other evaluate the predictive performance of the models in question. –  user10525 Jul 6 '12 at 13:03
I am comparing such different models just because we want to see all possible models. Our function just fits all models that are interesting to us - I know they are quite different in character :) In part, I also wanted to deliberately choose different models to see if even there the AIC could be used (if it could at all). –  Alexx Hardt Jul 6 '12 at 13:04
Why are you trying to convince them that AIC is not justified? –  John Jul 6 '12 at 13:44