Fisher's test with large data

Question

I want to compare the proportions in two samples, that can be organized as

               P      NP
SAMPLE_A  3,129,548 427,953
SAMPLE_B  2,930,639 407,353

If I look at the proportions I see that

               P        NP
SAMPLE_A      88%      12%
SAMPLE_B     87.8%    12.2%

which is a pretty good result, considering the kind of phenomenon under investigation.

However, since I'm dealing with millions of observations, both the Fisher's test and the Chi-Square approximation give p-values so low that I have to conclude that any difference of proportions that I observe is significant --- which is not: considering what I'm analyzing, a difference of 0.3% is not meaningful.

My question is: instead of simply show the similar proportions, can I do further analysis in order to better characterize such a similarity?

Answer 1

Your problem is that statistical significance is not the same as "everyday" significance, importance or relevance.

Your Fisher's test is statistically significant. That is, it's unlikely the (tiny) difference in proportions would have arisen by chance had the two populations been identical. As you write, this is in your case not due to a large difference in proportions, but to a large sample size. Any difference in populations, be it ever so tiny, will become statistically significant with sufficiently large sample size.

With smaller samples, statistical significance is associated with large effects. This association breaks down with large samples.

Your best approach will likely be not to look at statistical significance at all. You are interested in the actual difference between proportions, and you sound as if a difference of 0.2% is not a big deal. (There could of course be applications in which such a difference would be a big deal.)

Just communicate and discuss the raw proportions. Maybe visualize them using a mosaic plot. Don't even talk about p values or statistical significance.

Answer 2

Just to repeat something I just said in another answer: "But with sample sizes 7000 and 30000, there must be some substructure to the data, so I would worry about possible dependencies." Here your samplesize is in the millions, so it is difficult to believe there is no substructure to the data! So, please, tell us about how this data was obtained: If this is results from lab experiments, where there multiple labs? how long did it take? weeks? years? so, there could be drift of lab equipment with time, training effects (or boredom effect ...) with technicians, ...

You really need to tell us!