I have a very large sample (790 million rows), and in this case, even small difference are determined to be significant, p <0.0001. The 95% C.I.s are very sharp. In this case, these tests (t-test, chi square, and anova) show significant, but how meaningful is this significance ?
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ Have a look at Are large data sets inappropriate for hypothesis testing?. Similar question I suppose. $\endgroup$– rslCommented May 25, 2016 at 2:31
-
$\begingroup$ Read this paper: Using Effect Size—or Why the P Value Is Not Enough ncbi.nlm.nih.gov/pmc/articles/PMC3444174 $\endgroup$– Deep NorthCommented May 25, 2016 at 3:30
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
This is a good question I also encountered similar problem many times in real world. My own suggestion is NOT use hypothesis testing. If you really want to use somehting like that Maybe effect size?.
Here is my reason: the classical statistics framework assumes you have a very clean data. In most studies, you only have few of data points, but they come out from a carefully designed experiment. In such case, you will not have too many outliers and most data are some how satisfy coming form the Gaussian distribution assumption.
However, in the real world, like in your case, 790 million rows. I would bet there are many outliers and would not satisfy all the assumptions hypothesis testing framework is based on. Therefore it is not really meaningful to use the hypothesis testing framework.
Finally, let's assume your data clean. With such large sample size, you can think you have a really "powerful detector" by using hypothesis testing. This detector can tell you even a tinny differences between two populations.
For example, it is similar to check if on 2 persons' DNA, which is almost guaranteed to have a difference as a result. But you should ask yourself, does that "tiny difference" really matter to my business? Suppose, your business is trying to make uniforms based on person's height and weight, telling two populations are different on DNA level would not make too much difference on business decisions.
answered May 25, 2016 at 3:04
-
4$\begingroup$ You're right that with such a large dataset hypothesis testing often is not useful or applicable. However, this characterization of the "classical statistics framework" (involving "clean" data with a Gaussian distribution) is so narrow and limited it is almost unrecognizable. (Perhaps it was correct two centuries ago.) The implication that anything that doesn't fit these preconceptions could not be "meaningfully" tested does not appear to have any valid justification. $\endgroup$– whuber ♦Commented May 25, 2016 at 3:27