Given big enough sample size, a test will always show significant result unless the true effect size is exactly zero. Why?

Question

I am curious about a claim made in Wikipedia's article on effect size. Specifically:

[...] a non-null statistical comparison will always show a statistically significant results unless the population effect size is exactly zero

I am not sure what this means/implies, let alone an argument to back it up. I guess, after all, an effect is a statistic, i.e., a value calculated from a sample , with its own distribution. Does this mean that effects are never due to just random variation (which is what I understand it means to not be significant)? Do we then just consider whether the effect is strong enough -- having high absolute value?

I am considering the effect I am most familiar with: the Pearson correlation coefficient r seems to contradict this. Why would any $r$ be statistically-significant? If $r$ is small our regression line $$ y=ax+b = r\left(\frac {s_y}{s_x}\right) = \epsilon x+b $$

For $\epsilon$ small,is close to 0, an F-test will likely contain a confidence interval containing 0 for the slope. Isn't this a counterexample?

Answer 1

As @Kodiologist points out, this is really about what happens for large sample sizes. For small sample sizes there's no reason why you can't have false positives or false negatives.

I think the $z$-test makes the asymptotic case clearest. Suppose we have $X_1, \dots, X_n \stackrel{\text{iid}}\sim \mathcal N(\mu, 1)$ and we want to test $H_0: \mu = 0$ vs $H_A: \mu \neq 0$. Our test statistic is $$ Z_n = \frac{\bar X_n - 0}{1 / \sqrt n} = \sqrt n\bar X_n. $$

$\bar X_n \sim \mathcal N(\mu, \frac 1n)$ so $Z_n = \sqrt n \bar X_n \sim \mathcal N(\mu \sqrt n, 1)$. We are interested in $P(|Z_n| \geq \alpha)$. $$ P(|Z_n| \geq \alpha) = P(Z_n \leq -\alpha)+ P(Z_n \geq \alpha) $$ $$ = 1 + \Phi(-\alpha - \mu\sqrt n) - \Phi(\alpha - \mu \sqrt n). $$ Let $Y \sim \mathcal N(0,1)$ be our reference variable. Under $H_0$ $\mu = 0$ so we have $P(|Z_n| \geq \alpha) = 1 - P(-\alpha \leq Y \leq \alpha)$ so we can choose $\alpha$ to control our type I error rate as desired. But under $H_A$ $\mu \sqrt n \neq 0$ so $$ P(|Z_n| \geq \alpha) \to 1 + \Phi(\pm\infty) - \Phi(\pm\infty) = 1 $$ so with probability 1 we will reject $H_0$ if $\mu \neq 0$ (the $\pm$ is in case of $\mu < 0$, but either way the infinities have the same sign).

The point of this is that if $\mu$ exactly equals $0$ then our test statistic has the reference distribution and we'll reject 5% (or whatever we choose) of the time. But if $\mu$ is not exactly $0$, then the probability that we'll reject heads to $1$ as $n$ increases. The idea here is the consistency of a test, which is that under $H_A$ the power (probability of rejecting) heads to $1$ as $n \to \infty$.

It's the exact same story with the test statistic for testing $H_0 : \rho = \rho_0$ versus $H_A: \rho \neq \rho_0$ with the Pearson correlation coefficient. If the null hypothesis is false, then our test statistic gets larger and larger in probability, so the probability that we'll reject approaches $1$.

Answer 2

As a simple example, suppose that I am estimating your height using some statistical mumbo jumbo.

You've always stated to others that you are 177 cm (about 5 ft 10 in).

If I were to test this hypothesis (that your height is equal to 177 cm, $h = 177$), and I could reduce the error in my measurement enough, then I could prove that you are not in fact 177 cm. Eventually, if I estimate your height to enough decimal places, you would almost surely deviate from the stated height of 177.00000000 cm. Perhaps you are 177.02 cm; I only have to reduce my error to less than .02 to find out that you are not 177 cm.

How do I reduce the error in statistics? Get a bigger sample. If you get a large enough sample, the error gets so small that you can detect the most minuscule deviations from the null hypothesis.

Answer 3

Arguably what they said is wrong, if for no other reason than their use of "this always happens".

I don't know if this is the crux of the confusion you're having, but I'll post it because I think many do and will get confused by this:

"$X$ happens if $n$ is large enough" does NOT mean "If $n > n_0$, then $X$."

Rather, it means $\lim\limits_{n\to\infty} \Pr (X) = 1$.

What they are literally saying translates to the following:

For any sample size $n$ above some minimum size $n_0$, the result of any non-null test is guaranteed to be significant if the true effect size is not exactly zero.

What they were trying to say, though, is the following:

For any significance level, as the sample size is increased, the probability that a non-null test yields a significant result approaches 1 if the true effect size is not exactly zero.

There are crucial differences here:

• There is no guarantee. You are only more likely to get a significant result with a bigger sample. Now, they could dodge part of the blame here, because so far it's just a terminology issue. In a probabilistic context, it is understood that the statement "if n is large enough then X" can also be interpreted to mean "X becomes more and more likely to be true as n grows large".
  However, this interpretation goes out my window as soon as they say this "always" happens. The proper terminology here would have been to say this happens "with high probability"¹.

• This is secondary, but their wording is confusing—it seems to imply that you fix the sample size to be "large enough", and then the statement holds true for any significance level. However, regardless of what the precise mathematical statement is, that doesn't really make sense: you always first fix the significance level, and then you choose the sample size to be large enough.
  But the suggestion that it can somehow be the other way around unfortunately emphasizes the $n > n_0$ interpretation of "large enough", so that makes the above problem even worse.

But once you understand the literature, you get what they're trying to say.

(Side note: incidentally, this is exactly one of the constant problems many people have with Wikipedia. Frequently, it's only possible to understand what they're saying if you already know the material, so it's only good for a reference or as a reminder, not as self-teaching material.)

_{¹ For the fellow pedants (hi!), yes, the term has a more specific meaning than the one I linked to. The loosest technical term we probably want here is "asymptotically almost surely". See here.}

Answer 4

My favorite example is number of fingers by gender. The vast majority of people have 10 fingers. Some have lost fingers due to accidents. Some have extra fingers.

I don't know if men have more fingers than women (on average). All the easily available evidence suggests that men and women both have 10 fingers.

However, I am highly confident that if I did a census of all men and all women then I would learn that one gender has more fingers (on average) than the other.