Given big enough sample size, a test will always show significant result unless the true effect size is exactly zero. Why?

I am curious about a claim made in Wikipedia's article on effect size. Specifically:

[...] a non-null statistical comparison will always show a statistically significant results unless the population effect size is exactly zero

I am not sure what this means/implies, let alone an argument to back it up. I guess, after all, an effect is a statistic, i.e., a value calculated from a sample , with its own distribution. Does this mean that effects are never due to just random variation (which is what I understand it means to not be significant)? Do we then just consider whether the effect is strong enough -- having high absolute value?

I am considering the effect I am most familiar with: the Pearson correlation coefficient r seems to contradict this. Why would any $r$ be statistically-significant? If $r$ is small our regression line $$y=ax+b = r\left(\frac {s_y}{s_x}\right) = \epsilon x+b$$

For $\epsilon$ small,is close to 0, an F-test will likely contain a confidence interval containing 0 for the slope. Isn't this a counterexample?

• Hint: the clause before the portion you quoted is essential. "Given a sufficiently large sample size, a non-null statistical comparison will always show a statistically significant results unless the population effect size is exactly zero…" – Kodiologist Jan 19 '18 at 2:29
• @Kodiologist: But, re my example, would this imply that if sample size were bigger, then r itself would also be bigger, or, at least the expression $r(s_y/s_x)$ would be larger if sample size were larger? I don't see it. – gary Jan 19 '18 at 2:38
• If this wasn't true, it would be a flaw in the statistical method. If $\mu > \mu_0$, surely some sample size is large enough to detect the difference. – John Coleman Jan 19 '18 at 12:01

As a simple example, suppose that I am estimating your height using some statistical mumbo jumbo.

You've always stated to others that you are 177 cm (about 5 ft 10 in).

If I were to test this hypothesis (that your height is equal to 177 cm, $h = 177$), and I could reduce the error in my measurement enough, then I could prove that you are not in fact 177 cm. Eventually, if I estimate your height to enough decimal places, you would almost surely deviate from the stated height of 177.00000000 cm. Perhaps you are 177.02 cm; I only have to reduce my error to less than .02 to find out that you are not 177 cm.

How do I reduce the error in statistics? Get a bigger sample. If you get a large enough sample, the error gets so small that you can detect the most minuscule deviations from the null hypothesis.

• This is a very clear and concise explanation. It probably is more helpful for understanding why this happens than the more mathematical answers. Well done. – Nobody Jan 19 '18 at 18:15
• Nicely explained, but I think it's also important to consider that there are cases in which the stated value is truly exact. For example, setting aside weird things that happen in string theory etc., a measurement of the number of spatial dimensions of our universe (which can be done) is going to give 3, and no matter how precise you make that measurement, you will never consistently find statistically significant deviations from 3. Of course if you keep testing enough times you'll get some deviations simply due to variance, but that's a different issue. – David Z Jan 21 '18 at 6:34
• Probably a naive question but if I claim I am 177cm, doesn't the concept of significant digits mean I am only saying I am between 176.5 and 177.5? The answer seems to give a good theoretical concept, true, but is it not based on a false premise? What am I missing? – JimLohse Jan 31 '18 at 15:10
• In this case the stated height of 177 is analogous to the null hypothesis in statistics. In traditional hypothesis testing for equality, you make a statement of equality (e.g. $\mu = 177$). The point is that no matter what you state your height to be, I can disprove it by reducing the error unless the null hypothesis is EXACTLY true. I used height as an easy to understand example, but this concept is the same in other areas (substance x does not cause cancer, this coin is fair, etc.) – Underminer Jan 31 '18 at 16:47
• Mathematically, how is your claim "if you get a large enough sample, the error gets smaller..." supported? We compute the p-value from the $Z$ score, but I don't see how the Z score changes with sample size. – roulette01 Sep 10 at 15:07

As @Kodiologist points out, this is really about what happens for large sample sizes. For small sample sizes there's no reason why you can't have false positives or false negatives.

I think the $z$-test makes the asymptotic case clearest. Suppose we have $X_1, \dots, X_n \stackrel{\text{iid}}\sim \mathcal N(\mu, 1)$ and we want to test $H_0: \mu = 0$ vs $H_A: \mu \neq 0$. Our test statistic is $$Z_n = \frac{\bar X_n - 0}{1 / \sqrt n} = \sqrt n\bar X_n.$$

$\bar X_n \sim \mathcal N(\mu, \frac 1n)$ so $Z_n = \sqrt n \bar X_n \sim \mathcal N(\mu \sqrt n, 1)$. We are interested in $P(|Z_n| \geq \alpha)$. $$P(|Z_n| \geq \alpha) = P(Z_n \leq -\alpha)+ P(Z_n \geq \alpha)$$ $$= 1 + \Phi(-\alpha - \mu\sqrt n) - \Phi(\alpha - \mu \sqrt n).$$ Let $Y \sim \mathcal N(0,1)$ be our reference variable. Under $H_0$ $\mu = 0$ so we have $P(|Z_n| \geq \alpha) = 1 - P(-\alpha \leq Y \leq \alpha)$ so we can choose $\alpha$ to control our type I error rate as desired. But under $H_A$ $\mu \sqrt n \neq 0$ so $$P(|Z_n| \geq \alpha) \to 1 + \Phi(\pm\infty) - \Phi(\pm\infty) = 1$$ so with probability 1 we will reject $H_0$ if $\mu \neq 0$ (the $\pm$ is in case of $\mu < 0$, but either way the infinities have the same sign).

The point of this is that if $\mu$ exactly equals $0$ then our test statistic has the reference distribution and we'll reject 5% (or whatever we choose) of the time. But if $\mu$ is not exactly $0$, then the probability that we'll reject heads to $1$ as $n$ increases. The idea here is the consistency of a test, which is that under $H_A$ the power (probability of rejecting) heads to $1$ as $n \to \infty$.

It's the exact same story with the test statistic for testing $H_0 : \rho = \rho_0$ versus $H_A: \rho \neq \rho_0$ with the Pearson correlation coefficient. If the null hypothesis is false, then our test statistic gets larger and larger in probability, so the probability that we'll reject approaches $1$.

• Nitpick: if $μ < 0$, then $Z_n$ will diverge to $-\infty$ instead of $\infty$, right? – Kodiologist Jan 19 '18 at 5:18
• Nice, but what happens in the $\mu=0$ case should depend on whether $\bar{X}\to_p 0$ “faster” than $\sqrt{n}\to \infty$, right? I’m not even sure how you would “compare” the rate of convergence for a sequence of random variables and a sequence of integers - probably Slutsky’s theorem or something like that should be applied. – DeltaIV Jan 19 '18 at 7:37
• @DeltaIV, right, if the convergence rate were different, one would need a different scaling to get a nondegenerate null distribution. But for the present example, root-n is the right rate. – Christoph Hanck Jan 19 '18 at 9:13
• $\sqrt n \bar X$ converges to a standard normal by the CLT, not to $0$. – guy Jan 19 '18 at 14:27

Arguably what they said is wrong, if for no other reason than their use of "this always happens".

I don't know if this is the crux of the confusion you're having, but I'll post it because I think many do and will get confused by this:

Rather, it means $\lim\limits_{n\to\infty} \Pr (X) = 1$.

What they are literally saying translates to the following:

For any sample size $n$ above some minimum size $n_0$, the result of any non-null test is guaranteed to be significant if the true effect size is not exactly zero.

What they were trying to say, though, is the following:

For any significance level, as the sample size is increased, the probability that a non-null test yields a significant result approaches 1 if the true effect size is not exactly zero.

There are crucial differences here:

• There is no guarantee. You are only more likely to get a significant result with a bigger sample. Now, they could dodge part of the blame here, because so far it's just a terminology issue. In a probabilistic context, it is understood that the statement "if n is large enough then X" can also be interpreted to mean "X becomes more and more likely to be true as n grows large".
However, this interpretation goes out my window as soon as they say this "always" happens. The proper terminology here would have been to say this happens "with high probability"1.

• This is secondary, but their wording is confusing—it seems to imply that you fix the sample size to be "large enough", and then the statement holds true for any significance level. However, regardless of what the precise mathematical statement is, that doesn't really make sense: you always first fix the significance level, and then you choose the sample size to be large enough.
But the suggestion that it can somehow be the other way around unfortunately emphasizes the $n > n_0$ interpretation of "large enough", so that makes the above problem even worse.

But once you understand the literature, you get what they're trying to say.

(Side note: incidentally, this is exactly one of the constant problems many people have with Wikipedia. Frequently, it's only possible to understand what they're saying if you already know the material, so it's only good for a reference or as a reminder, not as self-teaching material.)

1 For the fellow pedants (hi!), yes, the term has a more specific meaning than the one I linked to. The loosest technical term we probably want here is "asymptotically almost surely". See here.

• "the probability that a non-null test yields a significant result approaches 0 if the true effect size is exactly zero" may not be quite right: if the test has significance level $\alpha$ then the probability of yielding a significant result may be $\alpha$ or thereabouts at all sample sizes – Henry Jan 19 '18 at 10:41
• @Henry: Oh shoot, you're right! I wrote it so fast I didn't stop to think. Thanks a ton! I've fixed it. :) – user541686 Jan 19 '18 at 11:16

My favorite example is number of fingers by gender. The vast majority of people have 10 fingers. Some have lost fingers due to accidents. Some have extra fingers.

I don't know if men have more fingers than women (on average). All the easily available evidence suggests that men and women both have 10 fingers.

However, I am highly confident that if I did a census of all men and all women then I would learn that one gender has more fingers (on average) than the other.