Is it okay to compare fitted distributions with the AIC? suppose I have a data set $x_1, \ldots, x_n$ and I would fit a normal, an exponential and a uniform distribution to them. The fitting function spits out a bunch of goodness-of-fit statistics, e.g. the AIC, BIC, chi-square, Kolmogorov-Smirnov, etc.
I am trying to convince someone that the AIC is not appropriate here, because we have different log-likelihoods, and sometimes different number of parameters, depending on the distributions. I would prefer the p-value of the Kolmogorov-Smirnov-Test to compare the fits.
Is my approach justified? How can I convince my coworker the AIC is not okay here (he likes to see a cited paper or something equivalent)?
edit: Specifically, I was shown this article: http://www.vosesoftware.com/whitepapers/Fitting%20distributions%20to%20data.pdf
I have no idea what to say to this. Page 4 lists the flaws of the chi-squared, Kolmogorov-Smirnov etc, and page 5 and 6 praise the AIC. Is he right?
 A: You have to penalize the model for number of parameters.  Let's say you had 30 data points and a model to fit it to that takes 29 parameters to define it.  You could fit the data perfectly.  But, that's not a terribly fair way to compare it to a uniform distribution with far fewer parameters.
The paper you cite mentions this.  Likely you're having trouble making an argument against it because there isn't a good general one.  The argument would be against how much you penalize for extra parameters in the model.  In that case you may want to examine different kinds of information criteria.
Furthermore, it's also a good idea to look at some other fit measures as well.  There's nothing wrong with using multiple ones and making rational arguments when the AIC differences are very small.
A: I'd go further and say it is probably the most widely accepted method for comparing distributions. But you should really use the corrected AIC, which has a piece added to it to adjust for small sample sizes. See Burnham and Anderson 2002, for example.
This site will take a set of numbers and do the comparisons for you, using the corrected AIC mentioned above.
http://www.easydatascience.com/
