In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically significant effect.".

Well, I for one didn't know that. Is this true? Does it mean that hypothesis testing is worthless for large data sets?

  • 10
    +1: this question usually exposes some interesting point of views. – user603 Sep 9 '10 at 19:05
  • 7
    More discussion on large data sets appears at stats.stackexchange.com/q/7815/919 . (The focus is on regression modeling there.) – whuber Mar 7 '11 at 16:01
  • 1
    related thread ? – Antoine Aug 29 '15 at 19:28
  • 8
    If a large sample makes you think hypothesis testing was the wrong tool, then hypothesis testing wasn't actually answering the right question at smaller samples either -- that it was wrong just became more obvious at large sample sizes, but the same considerations are relevant. If a significant result at a very small effect size makes you say "well, that's not what I wanted, I wanted it to tell me if it was important" then hypothesis testing was just the wrong tool to begin with. There are more suitable tools (e.g. confidence intervals, equivalence tests, etc) for that kind of problem. – Glen_b May 27 '16 at 1:09

14 Answers 14

up vote 88 down vote accepted

It is not true. If the null hypothesis is true then it will not be rejected more frequently at large sample sizes than small. There is an erroneous rejection rate that's usually set to 0.05 (alpha) but it is independent of sample size. Therefore, taken literally the statement is false. Nevertheless, it's possible that in some situations (even whole fields) all nulls are false and therefore all will be rejected if N is high enough. But is this a bad thing?

What is true is that trivially small effects can be found to be "significant" with very large sample sizes. That does not suggest that you shouldn't have such large samples sizes. What it means is that the way you interpret your finding is dependent upon the effect size and sensitivity of the test. If you have a very small effect size and highly sensitive test you have to recognize that the statistically significant finding may not be meaningful or useful.

Given some people don't believe that a test of the null hypothesis, when the null is true, always has an error rate equal to the cutoff point selected for any sample size, here's a simple simulation in R proving the point. Make N as large as you like and the rate of Type I errors will remain constant.

# number of subjects in each condition
n <- 100
# number of replications of the study in order to check the Type I error rate
nsamp <- 10000

ps <- replicate(nsamp, {
    #population mean = 0, sd = 1 for both samples, therefore, no real effect
    y1 <- rnorm(n, 0, 1) 
    y2 <- rnorm(n, 0, 1)
    tt <- t.test(y1, y2, var.equal = TRUE)
    tt$p.value
})
sum(ps < .05) / nsamp

# ~ .05 no matter how big n is. Note particularly that it is not an increasing value always finding effects when n is very large.
  • 8
    +1: indeed, all three answers here are logically consistent with one another. – user603 Sep 9 '10 at 19:03
  • 1
    Finally I've found a debunking of something a (non-statistics) professor told me a long time ago. – Jase Dec 15 '12 at 11:03
  • 1
    @Sympa, no. Just because SE goes down as N goes up doesn't mean you'll always find an effect with large N (see simulation). Keep in mind that as the SE is going down the quality of the estimate of the effect is increasing. If there's no population effect then it's much more likely to be close to 0 and show no difference. In fact, the distribution of p-values is flat regardless of sample size whenever the null is true (write your own simulation for that). There's no contradiction in the answer. – John Jun 20 '17 at 21:00
  • 4
    Then you would be wrong. You might want to consider reading the other answers here as well. Since you can't follow the relationship between the simulation and hypothesis testing, I guess I can only point to your primary claim that as standard error goes down, t goes up, and p goes down. This is only true should the effect stay constant. But the effect is a random sample and when the true effect is 0 then, as N increases the observed effect tends to decrease. Therefore, even though, as N increases SE goes down it won't increase t-values because the numerator in the t-value will also be lower. – John Jun 23 '17 at 4:32
  • 1
    The fact that rnorm can't produce an irrational number is irrelevant in the example. Even if it doesn't draw precisely normal from a mean of 0 and sd of 1, it's the same not normal for both samples. The Type I error rate might be ever so slightly off of .05 but it should stay constant regardless of N. And it's not true of all simulations since I could have chosen a discrete one where this isn't an issue. (If you really wanted to raise an esoteric issue then you should have addressed pseudo randomness.) – John Jun 20 at 10:53

I agree with the answers that have appeared, but would like to add that perhaps the question could be redirected. Whether to test a hypothesis or not is a research question that ought, at least in general, be independent of how much data one has. If you really need to test a hypothesis, do so, and don't be afraid of your ability to detect small effects. But first ask whether that's part of your research objectives.

Now for some quibbles:

  • Some null hypotheses are absolutely true by construction. When you're testing a pseudorandom number generator for equidistribution, for instance, and that PRG is truly equidistributed (which would be a mathematical theorem), then the null holds. Probably most of you can think of more interesting real-world examples arising from randomization in experiments where the treatment really does have no effect. (I would hold out the entire literature on esp as an example. ;-)

  • In a situation where a "simple" null is tested against a "compound" alternative, as in classic t-tests or z-tests, it typically takes a sample size proportional to $1/\epsilon^2$ to detect an effect size of $\epsilon$. There's a practical upper bound to this in any study, implying there's a practical lower bound on a detectable effect size. So, as a theoretical matter der Laan and Rose are correct, but we should take care in applying their conclusion.

  • Isn't all this a matter of type I error versus type II error (or power) ? If one fixes type I error probability ($\alpha$) at 0.05, then , obviously (except in the discrete case), it will be 0.05 whether the sample is large or not. But for a given type I error probability, 0.05 e.g., the power, or the probability that you will detect the effect when it is there, is larger for large sample sizes. – user83346 Aug 5 '17 at 11:34
  • @fcop Your comments, although correct, seem to be directed at other answers. They miss the point of this one, which is to suggest that not all statistical analyses need to be hypothesis tests. Type I and II errors have meaning only when conducting formal hypothesis tests. – whuber Aug 7 '17 at 14:57
  • the OP refers to a statement: ''We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically significant effect.'' So, if you test e.g. $H_0: \mu=1$ versus $H_1: \mu \ne 1$ then in large samples the power is so high that you 'detect' even small deviations from 1. So I think their statement is not correct, but that power in large samples allows you to detect very small differences. – user83346 Aug 7 '17 at 15:03
  • @fcop Thank you for explaining. I agree with your reasoning: when the null is true, then by construction even large studies will find a significant effect with a chance at most equal to the size of their test--that is, they will be unlikely to find a significant effect. – whuber Aug 7 '17 at 15:58

Hypothesis testing traditionally focused on p values to derive statistical significance when alpha is less than 0.05 has a major weakness. And, that is that with a large enough sample size any experiment can eventually reject the null hypothesis and detect trivially small differences that turn out to be statistically significant.

This is the reason why drug companies structure clinical trials to obtain FDA approval with very large samples. The large sample will reduce the standard error to close to zero. This in turn will artificially boost the t stat and commensurately lower the p value to close to 0%.

I gather within scientific communities that are not corrupted by economic incentives and related conflict of interest hypothesis testing is moving away from any p value measurements towards Effect Size measurements. This is because the unit of statistical distance or differentiation in Effect Size analysis is the standard deviation instead of the standard error. And, the standard deviation is completely independent from the sample size. The standard error on the other hand is totally dependent from the sample size.

So, anyone who is skeptical of hypothesis testing reaching statistically significant results based on large samples and p value related methodologies is right to be skeptical. They should rerun the analysis using the same data but using instead Effect Size statistical tests. And, then observe if the Effect Size is deemed material or not. By doing so, you could observe that a bunch of differences that are statistically significant are associated with Effect Size that are immaterial. That's what clinical trial researchers sometimes mean when a result is statistically significant but not "clinically significant." They mean by that one treatment may be better than placebo, but the difference is so marginal that it would make no difference to the patient within a clinical context.

  • 1
    One person's large sample is another's small sample. :) – Iterator Aug 4 '11 at 14:07
  • 2
    Didn't you ask the wrong question then? Maybe the FDA approval process should specify a larger gain vs placebo (perhaps related to costs of the drug, including its adverse effects) instead of just requiring statistical significance? Because there can very well be a real difference, albeit very small, and that difference was shown to be statistically significant however small it is. – Emil Vikström Jun 2 '14 at 15:58

A (frequentist) hypothesis test, precisely, address the question of the probability of the observed data or something more extreme would be likely assuming the null hypothesis is true. This interpretation is indifferent to sample size. That interpretation is valid whether the sample is of size 5 or 1,000,000.

An important caveat is that the test is only relevant to sampling errors. Any errors of measurement, sampling problems,coverage, data entry errors, etc are outside of the scope of sampling error. As sample size increases, non-sampling errors become more influential as small departures can produce significant departures from the random sampling model. As a result, tests of significance become less useful.

This is in no way an indictment of significance testing. However, we need to be careful about our attributions. A result may be statistically significant. However, we need to be cautious about how we make attributions when sample size is large. Is that difference due to our hypothesized generating process vis a vis sampling error or is it the result of any of a number of possible non-sampling errors that could influence the test statistic (which the statistic does not account for)?

Another consideration with large samples is the practical significance of a result. A significant test might suggest (even if we can rule out non-sampling error) a difference that is trivial in a practical sense. Even if that result is unlikely given the sampling model, is it significant in the context of the problem? Given a large enough sample, a difference in a few dollars might be enough to produce a result that is statistically significant when comparing income among two groups. Is this important in any meaningful sense? Statistical significance is no replacement for good judgment and subject matter knowledge.

As an aside, the null is neither true nor false. It is a model. It is an assumption. We assume the null is true and assess our sample in terms of that assumption. If our sample would be unlikely given this assumption, we place more trust in our alternative. To question whether or not a null is ever true in practice is a misunderstanding of the logic of significance testing.

  • 3
    This supports an argument for increased model complexity as sample sizes become large - in the large sample case sampling error is no longer the dominant sourse of uncertainty. Of course this only "makes sense" in a Bayesian framework, which allows for other sources of uncertainty besides sampling error. – probabilityislogic Apr 11 '14 at 23:06

One simple point not made directly in another answer is that it's simply not true that "all null hypotheses are false."

The simple hypothesis that a physical coin has heads probability exactly equal to 0.5, ok, that is false.

But the compound hypothesis that a physical coin has heads probability greater than 0.499 and less than 0.501 may be true. If so, no hypothesis test -- no matter how many coin flips go into it -- is going to be able to reject this hypothesis with a probability greater than $\alpha$ (the tests's bound on false positives).

The medical industry tests "non-inferiority" hypotheses all the time, for this reason -- e.g. a new cancer drug has to show that its patients' probability of progression-free survival isn't less than 3 percentage points lower than an existing drug's, at some confidence level (the $\alpha$, usually 0.05).

In a certain sense, [all] many null hypothesis are [always] false (the group of people living in houses with odd numbers does never exactly earn the same on average as the group of people living in houses with even numbers).

In the frequentist framework, the question that is asked is whether the difference in income between the two group is larger than $T_{\alpha}n^{-0.5}$ (where $T_{\alpha}$ is the $\alpha$ quantile of the distribution of the test statistic under the null). Obviously, for $n$ growing without bounds, this band becomes increasingly easy to break through.

This is not a defect of statistical tests. Simply a consequence of the fact that without further information (a prior) we have that a large number of small inconsistencies with the null have to be taken as evidence against the null. No matter how trivial these inconsistencies turn out to be.

In large studies, it becomes then interesting to re-frame the issue as a bayesian test, i.e. ask oneself (for instance), what is $\hat{P}(|\bar{\mu}_1-\bar{\mu}_2|^2>\eta|\eta, X)$.

  • That's weird... intuitively, this seems to contradict the Law of Large Numbers. – Carlos Accioly Sep 9 '10 at 19:12
  • Carlos:> can you be more specific ? – user603 Sep 9 '10 at 19:25
  • The LLN basically states that the larger your sample is, the better it represents the "real" probability distribution. In your example, the more house numbers I examine, the closer to 50% the number of odd-numbered houses will be. So it sounds weird that it becomes easier for you to break through the band, since it shrinks in proportion to the square root of $n$. (Am I making sense here?) – Carlos Accioly Sep 9 '10 at 19:54
  • 1
    @Carlos -- but convergence does not mean equality; this is guaranteed only for unreachable limit of infinity. So there is no contradiction ;-) – mbq Sep 9 '10 at 23:32

The short answer is "no". Research on hypothesis testing in the asymptotic regime of infinite observations and multiple hypotheses has been very, very active in the past 15-20 years, because of microarray data and financial data applications. The long answer is in the course page of Stat 329, "Large-Scale Simultaneous Inference", taught in 2010 by Brad Efron. A full chapter is devoted to large-scale hypothesis testing.

  • 7
    I believe that Efron's book focuses on a large number of variables (and the resulting multiple testing issues that arise), not sample size. – Galit Shmueli Aug 9 '11 at 14:17

Hypothesis testing for large data should the desired level of difference into account, rather than whether there is a difference or not. You're not interested in the H0 that the estimate is exactly 0. A general approach would be to test whether the difference between the null hypothesis and the observed value is larger than a given cut-off value.

An simple example with the T-test: You can make following assumptions for big sample sizes, given you have equal sample sizes and standard deviations in both groups, and $\bar{X_1} > \bar{X_2}$ : $$T=\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}+\frac{\delta}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$ hence $$T=\frac{\bar{X1}-\bar{X2}}{\sqrt{\frac{S^2}{n}}} \approx N(\frac{\delta}{\sqrt{\frac{S^2}{n}}},1)$$

as your null hypothesis $H_0:\bar{X1}-\bar{X2} = \delta $ implies:

$$\frac{\bar{X1}-\bar{X2}-\delta}{\sqrt{\frac{S^2}{n}}}\approx N(0,1)$$

This you can easily use to test for a significant and relevant difference. In R you can make use of the noncentrality parameter of the T distributions to generalize this result for smaller sample sizes as well. You should take into account that this is a one-sided test, the alternative $H_A$ is $\bar{X1}-\bar{X2} > \delta $.

mod.test <- function(x1,x2,dif,...){
    avg.x1 <- mean(x1)
    avg.x2 <- mean(x2)
    sd.x1 <- sd(x1)
    sd.x2 <- sd(x2)

    sd.comb <- sqrt((sd.x1^2+sd.x2^2)/2)
    n <- length(x1)
    t.val <- (abs(avg.x1-avg.x2))*sqrt(n)/sd.comb
    ncp <- (dif*sqrt(n)/sd.comb)
    p.val <- pt(t.val,n-1,ncp=ncp,lower.tail=FALSE)
    return(p.val)
}

n <- 5000

test1 <- replicate(100,
  t.test(rnorm(n),rnorm(n,0.05))$p.value)
table(test1<0.05)
test2 <- replicate(100,
  t.test(rnorm(n),rnorm(n,0.5))$p.value)
table(test2<0.05)

test3 <- replicate(100,
   mod.test(rnorm(n),rnorm(n,0.05),dif=0.3))
table(test3<0.05)

test4 <- replicate(100,
   mod.test(rnorm(n),rnorm(n,0.5),dif=0.3))
table(test4<0.05)

Which gives :

> table(test1<0.05)
FALSE  TRUE 
   24    76 

> table(test2<0.05)
TRUE 
 100 

> table(test3<0.05)
FALSE 
  100 

> table(test4<0.05)
TRUE 
 100 
  • isn't there a copy/past typo in the first equation ? – user603 Sep 10 '10 at 19:53
  • I don't see it? – Joris Meys Sep 12 '10 at 9:54

"Does it mean that hypothesis testing is worthless for large data sets?"

No, it doesn't mean that. The general message is that decisions made after conducting a hypothesis test should always take into account the estimated effect size, and not only the p-value. Particularly, in experiments with very large sample sizes, this necessity to consider the effect size becomes dramatic. Of course, in general, users don't like this because the procedure becomes less "automatic".

Consider this simulation example. Suppose you have a random sample of 1 million observations from a standard normal distribution,

n <- 10^6
x <- rnorm(n)

and another random sample of 1 million observations from a normal distribution with mean equal to $0.01$ and variance equal to one.

y <- rnorm(n, mean = 0.01)

Comparing the means of the two populations with a t-test at the canonical $95\%$ confidence level, we get a tiny p-value of approximately $2.5\times 10^{-14}$.

t.test(x, y)

        Welch Two Sample t-test

data:  x and y
t = -7.6218, df = 1999984, p-value = 2.503e-14
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
 -0.013554059 -0.008009031
sample estimates:
   mean of x    mean of y 
0.0008947038 0.0116762485

It's correct to say that the t-test "detected" that the means of the two populations are different. But take a look at the very short $95\%$ confidence interval for the difference between the two population means: $[-0.013, -0.008]$.

Is a difference between the two population means of this order of magnitude relevant to the particular problem we are studying or not?

  • I agree with everything in your answer except that first sentence, which I'd change to "Yes, it usually means that," because with large samples of a million or so the effect sizes are SO small. – zbicyclist Jun 20 '17 at 21:10
  • Isn't all this a matter of type I error versus type II error (or power) ? If one fixes type I error probability ($\alpha$) at 0.05, then , obviously (except in the discrete case), it will be 0.05 whether the sample is large or not. But for a given type I error probability, 0.05 e.g., the power, or the probability that you will detect the effect when it is there, is larger for large sample sizes. – user83346 Aug 5 '17 at 11:33

I think its a problem of most significance tests having some general undefined class of implicit alternatives to the null, which we never know. Often these classes may contain some sort of "sure thing" hypothesis, in which the data fits perfectly (i.e. a hypothesis of the form $H_{ST}:d_{1}=1.23,d_{2}=1.11,\dots$ where $d_{i}$ is the ith data point). The value of the log likelihood is such an example of a significance test which has this property.

But one is usually not interested in these sure thing hypothesis. If you think about what you actually want to do with the hypothesis test, you will soon recognise that you should only reject the null hypothesis if you have something better to replace it with. Even if your null does not explain the data, there is no use in throwing it out, unless you have a replacement. Now would you always replace the null with the "sure thing" hypothesis? Probably not, because you can't use these "sure thing" hypothesis to generalise beyond your data set. It's not much more than printing out your data.

So, what you should do is specify the hypothesis that you would actually be interested in acting on if they were true. Then do the appropriate test for comparing those alternatives to each other - and not to some irrelevant class of hypothesis which you know to be false or unusable.

Take the simple case of testing the normal mean. Now the true difference may be small, but adopting a position similar to that in @keith's answer, we simply test the mean at various discrete values that are of interest to us. So for example, we could have $H_{0}:\mu=0$ vs $H_{1}:\mu\in\{\pm 1,\pm 2,\pm 3,\pm 4,\pm 5,\pm 6\}$. The problem then transfers to looking at what level do we want to do these tests at. This has a relation to the idea of effect size: at what level of graininess would have an influence on your decision making? This may call for steps of size $0.5$ or $100$ or something else, depending on the meaning of the test and of the parameters. For instance if you were comparing the average wealth of two groups, would anyone care if there was a difference of two dollars, even if it was 10,000 standard errors away from zero? I know I wouldn't.

The conclusion is basically that you need to specify your hypothesis space - those hypothesis that you are actually interested in. It seems that with big data, this becomes a very important thing to do, simply because your data has so much resolving power. It also seems like it is important to compare like hypothesis - point with point, compound with compound - to get well behaved results.

No. It is true, that all useful point hypothesis tests are consistent and thus will show up a significant result if only the sample size is large enough and some irrelevant effect exists. To overcome this drawback of statistical hypotheses testing (already mentioned by the answer of Gaetan Lion above), there are relevance tests. These are similar to equivalence tests but even less common. For a relevance test, the size of a minimum relevant effect is prespecified. A relevance test can base on a confidence interval for the effect: If the confidence interval and the relevance region are disjoint, you may reject the null.

However, van der Laan and Rose assume in their statement, that even true null hypotheses are tested in studies. If a null hypothesis is true, the propability to reject is not larger than alpha, especially in the case of large samples and even misspecified I can only see that the sample distribution is systematically different from the population distribution,

The article you mention does have a valid point, as far as standard frequentist tests are concerned. That is why testing for a given effect size is very important. To illustrate, here is an anova between 3 groups, where group B slightly different than group A and C. try this in r:

treat_diff=0.001 #size of treatment difference
ns=c(10, 100, 1000, 10000, 100000, 1000000) #values for sample size per group considered
reps=10 #number of test repetitions for each sample size considered
p_mat=data.frame(n=factor(), p=double()) #create empty dataframe for outputs
for (n in ns){ #for each sample size
  for (i in c(1:reps)){ #repeat anova test ‘reps’ time
    treatA=data.frame(treatment="A", val=rnorm(n)) 
    treatB=data.frame(treatment="B", val=rnorm(n)+treat_diff) #this is the group that has the means slightly different from the other groups
    treatC=data.frame(treatment="C", val=rnorm(n))
    all_treatment=rbind(treatA, treatB, treatC)
    treatment_aov=aov(val~treatment, data=all_treatment)
    aov_summary=summary(treatment_aov)
    p=aov_summary[[1]][["Pr(>F)"]][1]
    temp_df=data.frame(n=n, p=p)
    p_mat=rbind(p_mat, temp_df)
  }
}

library(ggplot2)
p <- ggplot(p_mat, aes(factor(n), p))
p + geom_boxplot()

As expected, with greater number of samples per test, the statistical significance of the test increases: enter image description here

I think what they mean is that one often makes an assumption about the probability density of the null hypothesis which has a 'simple' form but does not correspond to the true probability density.

Now with small data sets, you might not have enough sensitivity to see this effect but with a large enough data set you will reject the null hypothesis and conclude that there is a new effect instead of concluding that your assumption about the null hypothesis is wrong.

  • 1
    I do not know if Mark and Shern had your view in mind but just to re-phrase your point- if the model for the data under the null is 'wrong' then you will reject the null hypothesis for a large enough data. – user28 Sep 10 '10 at 8:40

Isn't all this a matter of type I error versus type II error (or power) ? If one fixes the type I error probability ($\alpha$) at 0.05, then , obviously (except in the discrete case), it will be 0.05 whether the sample is large or not.

But for a given type I error probability, 0.05 e.g., the power, or the probability that you will detect the effect when it is there (so the probability to reject $H_0$ (=detect the effect) when $H_1$ is true (=when the effect is there)), is larger for large sample sizes.

Power increases with sample size (all other things equal).

But the statement that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of no effect is true — will declare a statistically significant effect." is incorrect.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.