Are large data sets inappropriate for hypothesis testing?

Question

In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically significant effect.".

Well, I for one didn't know that. Is this true? Does it mean that hypothesis testing is worthless for large data sets?

Answer 1

It is not true. If the sample size is large enough and the null hypothesis is true it will not be rejected. Nevertheless, it's possible that all nulls are false and therefore all will be rejected if N is high enough.

What is true is that trivially small effects are found with very large sample sizes. That in no way suggests you shouldn't have such large samples sizes. What it means is that the way you interpret your finding is dependent upon the effect size and sensitivity of the test. If you have a very small effect size and highly sensitive test you have to recognize that the statistically significant finding may not be meaningful or useful.

Answer 2

I agree with the answers that have appeared, but would like to add that perhaps the question could be redirected. Whether to test a hypothesis or not is a research question that ought, at least in general, be independent of how much data one has. If you really need to test a hypothesis, do so, and don't be afraid of your ability to detect small effects. But first ask whether that's part of your research objectives.

Now for some quibbles:

• Some null hypotheses are absolutely true by construction. When you're testing a pseudorandom number generator for equidistribution, for instance, and that PRG is truly equidistributed (which would be a mathematical theorem), then the null holds. Probably most of you can think of more interesting real-world examples arising from randomization in experiments where the treatment really does have no effect. (I would hold out the entire literature on esp as an example. ;-)

• In a situation where a "simple" null is tested against a "compound" alternative, as in classic t-tests or z-tests, it typically takes a sample size proportional to $1/\epsilon^2$ to detect an effect size of $\epsilon$. There's a practical upper bound to this in any study, implying there's a practical lower bound on a detectable effect size. So, as a theoretical matter der Laan and Rose are correct, but we should take care in applying their conclusion.

Answer 3

A (frequentist) hypothesis test, precisely, address the question of the probability of the observed data or something more extreme would be likely assuming the null hypothesis is true. This interpretation is indifferent to sample size. That interpretation is valid whether the sample is of size 5 or 1,000,000.

An important caveat is that the test is only relevant to sampling errors. Any errors of measurement, sampling problems,coverage, data entry errors, etc are outside of the scope of sampling error. As sample size increases, non-sampling errors become more influential as small departures can produce significant departures from the random sampling model. As a result, tests of significance become less useful.

This is in no way an indictment of significance testing. However, we need to be careful about our attributions. A result may be statistically significant. However, we need to be cautious about how we make attributions when sample size is large. Is that difference due to sampling error (which the test assumes) or is it the result of any of a number of possible non-sampling errors that could influence the test statistic?

Another consideration with large samples is the practical significance of a result. A significant test might suggest (even if we can rule out non-sampling error) a difference that is trivial in a practical sense. Even if that result is unlikely given the sampling model, is it significant in the context of the problem? Given a large enough sample, a difference in a few dollars might be enough to produce a result that is statistically significant when comparing income among two groups. Is this important in any meaningful sense? Statistical significance is no replacement for good judgment and subject matter knowledge.

As an aside, the null is neither true nor false. It is a model. It is an assumption. We assume the null is true and assess our sample in terms of that assumption. If our sample would be unlikely given this assumption, we place more trust in our alternative. To question whether or not a null is ever true in practice is a misunderstanding of the logic of significance testing.

Answer 4

The short answer is "no". Research on hypothesis testing in the asymptotic regime of infinite observations and multiple hypotheses has been very, very active in the past 15-20 years, because of microarray data and financial data applications. The long answer is in the course page of Stat 329, "Large-Scale Simultaneous Inference", taught in 2010 by Brad Efron. A full chapter is devoted to large-scale hypothesis testing.

Answer 5

In a certain sense, [all] many null hypothesis are [always] false (the group of people living in houses with odd numbers does never exactly earn the same on average as the group of people living in houses with even numbers).

In the frequentist framework, the question that is asked is whether the difference in income between the two group is larger than $T_{\alpha}n^{-0.5}$ (where $T_{\alpha}$ is the $\alpha$ quantile of the distribution of the test statistic under the null). Obviously, for $n$ growing without bounds, this band becomes increasingly easy to break through.

This is not a defect of statistical tests. Simply a consequence of the fact that without further information (a prior) we have that a large number of small inconsistencies with the null have to be taken as evidence against the null. No matter how trivial these inconsistencies turn out to be.

In large studies, it becomes then interesting to re-frame the issue as a bayesian test, i.e. ask oneself (for instance), what is $\hat{P}(|\bar{\mu}_1-\bar{\mu}_2|^2>\eta|\eta, X)$.

Answer 6

Hypothesis testing traditionally focused on p values to derive statistical significance when alpha is less than 0.05 has a major weakness. And, that is that with a large enough sample size any experiment can eventually reject the null hypothesis and detect trivially small differences that turn out to be statistically significant.

This is the reason why drug companies structure clinical trials to obtain FDA approval with very large samples. The large sample will reduce the standard error to close to zero. This in turn will artificially boost the t stat and commensurately lower the p value to close to 0%.

I gather within scientific communities that are not corrupted by economic incentives and related conflict of interest hypothesis testing is moving away from any p value measurements towards Effect Size measurements. This is because the unit of statistical distance or differentiation in Effect Size analysis is the standard deviation instead of the standard error. And, the standard deviation is completely independent from the sample size. The standard error on the other hand is totally dependent from the sample size.

So, anyone who is skeptical of hypothesis testing reaching statistically significant results based on large samples and p value related methodologies is right to be skeptical. They should rerun the analysis using the same data but using instead Effect Size statistical tests. And, then observe if the Effect Size is deemed material or not. By doing so, you could observe that a bunch of differences that are statistically significant are associated with Effect Size that are immaterial. That's what clinical trial researchers sometimes mean when a result is statistically significant but not "clinically significant." They mean by that one treatment may be better than placebo, but the difference is so marginal that it would make no difference to the patient within a clinical context.

Answer 7

I think what they mean is that one often makes an assumption about the probability density of the null hypothesis which has a 'simple' form but does not correspond to the true probability density.

Now with small data sets, you might not have enough sensitivity to see this effect but with a large enough data set you will reject the null hypothesis and conclude that there is a new effect instead of concluding that your assumption about the null hypothesis is wrong.

Answer 8

Hypothesis testing for large data should the desired level of difference into account, rather than whether there is a difference or not. You're not interested in the H0 that the estimate is exactly 0. A general approach would be to test whether the difference between the null hypothesis and the observed value is larger than a given cut-off value.

An simple example with the T-test: You can make following assumptions for big sample sizes, given you have equal sample sizes and standard deviations in both groups, and $\bar{X_1} > \bar{X_2}$ : $$T=\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}+\frac{\delta}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$ hence $$T=\frac{\bar{X1}-\bar{X2}}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$

as your null hypothesis $H_0:\bar{X1}-\bar{X2} = \delta $ implies:

$$\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}\approx N(0,1)$$

This you can easily use to test for a significant and relevant difference. In R you can make use of the noncentrality parameter of the T distributions to generalize this result for smaller sample sizes as well. You should take into account that this is a one-sided test, the alternative $H_A$ is $\bar{X1}-\bar{X2} > \delta $.

mod.test <- function(x1,x2,dif,...){
    avg.x1 <- mean(x1)
    avg.x2 <- mean(x2)
    sd.x1 <- sd(x1)
    sd.x2 <- sd(x2)

    sd.comb <- sqrt((sd.x1^2+sd.x2^2)/2)
    n <- length(x1)
    t.val <- (abs(avg.x1-avg.x2))*sqrt(n)/sd.comb
    ncp <- (dif*sqrt(n)/sd.comb)
    p.val <- pt(t.val,n-1,ncp=ncp,lower.tail=FALSE)
    return(p.val)
}

n <- 5000

test1 <- replicate(100,
  t.test(rnorm(n),rnorm(n,0.05))$p.value)
table(test1<0.05)
test2 <- replicate(100,
  t.test(rnorm(n),rnorm(n,0.5))$p.value)
table(test2<0.05)

test3 <- replicate(100,
   mod.test(rnorm(n),rnorm(n,0.05),dif=0.3))
table(test3<0.05)

test4 <- replicate(100,
   mod.test(rnorm(n),rnorm(n,0.5),dif=0.3))
table(test4<0.05)

Which gives :

> table(test1<0.05)
FALSE  TRUE 
   24    76 

> table(test2<0.05)
TRUE 
 100 

> table(test3<0.05)
FALSE 
  100 

> table(test4<0.05)
TRUE 
 100 

Answer 9

One simple point not made directly in another answer is that it's simply not true that "all null hypotheses are false."

The simple hypothesis that a physical coin has heads probability exactly equal to 0.5, ok, that is false.

But the compound hypothesis that a physical coin has heads probability greater than 0.499 and less than 0.501 may be true. If so, no hypothesis test -- no matter how many coin flips go into it -- is going to be able to reject this hypothesis with a probability greater than $\alpha$ (the tests's bound on false positives).

The medical industry tests "non-inferiority" hypotheses all the time, for this reason -- e.g. a new cancer drug has to show that its patients' probability of progression-free survival isn't less than 3 percentage points lower than an existing drug's, at some confidence level (the $\alpha$, usually 0.05).

Answer 10

I think its a problem of most significance tests having some general undefined class of implicit alternatives to the null, which we never know. Often these classes may contain some sort of "sure thing" hypothesis, in which the data fits perfectly (i.e. a hypothesis of the form $H_{ST}:d_{1}=1.23,d_{2}=1.11,\dots$ where $d_{i}$ is the ith data point). The value of the log likelihood is such an example of a significance test which has this property.

But one is usually not interested in these sure thing hypothesis. If you think about what you actually want to do with the hypothesis test, you will soon recognise that you should only reject the null hypothesis if you have something better to replace it with. Even if your null does not explain the data, there is no use in throwing it out, unless you have a replacement. Now would you always replace the null with the "sure thing" hypothesis? Probably not, because you can't use these "sure thing" hypothesis to generalise beyond your data set. It's not much more than printing out your data.

So, what you should do is specify the hypothesis that you would actually be interested in acting on if they were true. Then do the appropriate test for comparing those alternatives to each other - and not to some irrelevant class of hypothesis which you know to be false or unusable.

Take the simple case of testing the normal mean. Now the true difference may be small, but adopting a position similar to that in @keith's answer, we simply test the mean at various discrete values that are of interest to us. So for example, we could have $H_{0}:\mu=0$ vs $H_{1}:\mu\in\{\pm 1,\pm 2,\pm 3,\pm 4,\pm 5,\pm 6\}$. The problem then transfers to looking at what level do we want to do these tests at. This has a relation to the idea of effect size: at what level of graininess would have an influence on your decision making? This may call for steps of size $0.5$ or $100$ or something else, depending on the meaning of the test and of the parameters. For instance if you were comparing the average wealth of two groups, would anyone care if there was a difference of two dollars, even if it was 10,000 standard errors away from zero? I know I wouldn't.

The conclusion is basically that you need to specify your hypothesis space - those hypothesis that you are actually interested in. It seems that with big data, this becomes a very important thing to do, simply because your data has so much resolving power. It also seems like it is important to compare like hypothesis - point with point, compound with compound - to get well behaved results.