About goodness-of-fit p-value and Pearson's $\chi^2$ statistic

Question

In a simulation, I generate data from distribution A, and then I hope to test if each of some other distributions B,C,D... can fit the data equally well. Obviously there is an initial fit of the model under distribution A, so I can get the Pearson's chi^2 statistic as a measure of goodness-of-fit.

My question: is it possible that under distributions B,C,D... I may get a even smaller chi^2 statistic than the one obtained from the true model (i.e. distn A)? The reason for this question is that I am considering whether a 1 or 2-sided test should be done, i.e. H0: model A fits the data; Ha: not H0.

Intuitively (maybe I am wrong), a more general model (I mean, more general than A, such as A=Poisson, B=negative binomial) might provide an equally well or better fit for the data generated from A, so the Pearson's statistic is expected to be smaller if I use distn B to model data simulated from distn A. In this case, I should get a large p-value indicating no lack-of-fit, right? Any suggestions on choosing alternative statistics for evaluating GOF?

Thank you very much!

Answer 1

I don't see a true connection to one-sided tests. First let's be sure we understand what a goodness-of-fit does. We start with a hypothesized distribution. In the real world there is no exact distribution that the data is generated from. The distribution is a model. The purpose of the goodness of fit test is to determine whether or not the data suggests that our model is wrong. So an issue in goodness of fit is how big a departure from the hypothesized model matters for my practical purposes. So the sample size should be chosen to have good power to reject the null hypothesis when the difference is large enough to matter to us. This is not a very precise notion but I think that when we do data analysis we have a sense of when the data is far from an assumed normal population for example. We understand this when we look at parameters such as skewness and kurtosis or we see the departutes on a qq plot.

When we use chi-square we do not have an exact test. So the significance level of the test is only approximate. This will be a problem when the sample size is small (in addition to possible lack of power).

I think when you seriously consider these facts it should not be surprising that when you artificially sample from a particular distribution and pick a similar (but different distribution) to test a given sample could easily lead to a smaller or larger chi-square value then if the "true" distribution were hypothesized for the fit. Random variability, estimation of unknown parameters, sample size, approximation of the chi-square test statistic are all factors that enter into this.

Answer 2

I think it depends on what you mean by "different distribution". E.g if you generate data that is $\sim(\mathcal{N}(0,10)$ and compare it to a distribution such as $\sim(\mathcal{N(0.02, 10)}$ then sure, in some cases the second distribution will fit better, just by chance, and will almost certainly fit just about as well.