Understanding why a $p$-value is too small I have a data set with counts of particles, and I want to test if they follow a distribution. For a certain species, I make the $\chi^2$-test, and everything seems reasonable, finding a $p$-value of $p=0.75$, which I interpret as meaning that my null hypothesis (in this case, that the data follows a Skellam distribution) is not rejected. In the plot below, the histogram is the binned data and the curve is the expected distribution.

However, when I do the same for another type of particle, I find the $p$-value of $p=2\times 10^{-13}$, but the plot suggests to me that the data does follow the Skellam distribution:

I read in some places that a large sample may cause this, but my sample size is $3\times10^4$, and as it didn't affect the first type of particles, I'm assuming that is not the problem. What am I misunderstanding here? Does it really mean that I should reject the null hypothesis?
A follow-up question: which test should I use to check the goodness-of-fit, in this case? For completeness, the statistic is $\chi^2=19.8$ and $\chi^2=220$, respectively.
 A: A few general thoughts:


*

*It's very rare that real-world data follow a specific distribution exactly. This doesn't stop us from using a specific distribution as a model in order to answer questions. A model doesn't have to be perfect, but good enough for the purpose.

*With such a huge sample size, even tiny deviations from a Skellam distribution will result in very small p-values. This is just a result of the consistency of the tests. The power to detect smaller and smaller deviations increases with increasing sample size (see also here). In the second case, a p-value of $2\times 10^{-13}$ means that there is a lot of evidence against the null hypothesis that the data come from a Skellam distribution. Specifically, there are $-\log_{2}(2\times 10^{-13})\approx 42.19$ bits of information agains the test hypothesis (this is called the $S$-value).

*A failure to reject the null hypothesis does not mean that it is true. It means that there is not enough evidence to reject the notion that your data are compatible with a Skellam distribution with a high enough confidence. There may be infinitely many distributions other than the Skellam distribution that are compatible with your data.

*Histograms are suboptimal to check the agreement between the data and a specified distribution. I recommend to use Q-Q-plots instead (more information here). Another very useful visualization tool is the hanging rootogram. A good paper on this can be found here. I show how to apply a hanging rootogram to check the fit to a Poisson distribution in this answer.


In the light of the points above, here are some questions that you might find useful to ask yourself:


*

*What's your specific goal? Why do you want to show that the data are following a Skellam distribution?

*How large do the deviations from a Skellam distribution have to be in order for you to deem the model of a Skellam distribution unsuitable for the task?


Both of these questions require subject matter knowledge which I don't have.
