# Why does the Pearson chi-square test require to reject the hypothesis?

I generate $$100000$$ independent random numbers with equal probabilities $$\frac{1}{35}$$: $$\log_2(1),\log_2(2),...,\log_2(35)$$.
Then I summarize $$25$$ neighboring values:
$$S_1=k_1+k_2+...+k_{25}$$
$$S_2=k_2+k_3+...+k_{26}$$
$$S_3=k_3+k_4+...+k_{27}$$ etc.

Next, I test the distribution of $$S$$ for normality with Pearson's test. Using empirical $$S_i$$, I find the expectation and variance: $$E(S)=3.80, var(S)=0.24$$. Grouping $$S_i$$ by 17 intervals, I calculate the value of chi-square statistics, that is about 978. It is much more than the critical value of chi-square test. So I would be forced to reject the hypothesis about normality.

However, I can graphically see that the distribution of $$S$$ is really close to Normal. Why does the Pearson chi-square test require to reject the hypothesis?

• Sums such as your $S_i$ may be approximately (but not exactly) normally distributed. In such instances a histogram may seem to fit a normal distribution fairly well. But judging normality by eye from a histogram can be risky. Histograms can be be made in various ways, some better for judging normality than others. [Also, other kinds of graphs may be even more useful.] But formal tests of normality (e.g., chi-squared) may be able to distinguish btw. (i) a sample that seems to have come from a normal distribution, but did not, and (ii) a sample that truly did come from a normal distribution. Commented Feb 1, 2021 at 22:02

## 2 Answers

It’s really close to normal. In other words, it isn’t normal. The test has enough observations to detect that slight difference, and the p-value is behaving as advertised.

The usual sorts of hypothesis test, like those one meets in a basic course on statistics, will not answer the question "is it visually similar" but something much more like "is it different enough for us to be able to tell that the difference is not merely due to sampling variation".

In large samples a test may be able to detect pretty small departures even if they were practically inconsequential. Indeed differences considerably smaller than a single pixel in such a plot as the one you show, might be easily detected when a test is conducted at sufficiently large sample sizes.

The distinction between those two questions is critical - not just in goodness of fit, but that's where the apparent confusion most often arises. It is essentially the distinction between 'practical' and 'statistical' significance.

This discrepancy is not the fault of hypothesis testing; it's in the choice to use it in a situation where your actual question of interest is not the one answered by the test that is chosen.

There are some kinds of tests that can be used for questions like "are these two things sufficiently close together for some purpose" (as long as you can define what you mean sufficiently precisely to turn that into a specific numerical value on a specific measure of closeness for some specific case; e.g. see equivalence tests on means), but in many cases I've encountered, I'd argue that in general if the question you're really interested in "are these two things very alike", a test probably isn't called for at all.

One thing to keep in mind is that at very large sample sizes issues like bias may start to loom much larger than sampling variation even when those biases are tiny, so statistical significance may be uninformative in such cases - e.g. consider that even a very small net bias from all sources could be large enough to detect, rather than the null actually being false in the population.