What happened to statistical significance in regression when data size is gigantic?

I was reading this question regarding large scale regression (link) where whuber pointed out an interesting point as follows:

"Almost any statistical test you run will be so powerful that it's almost sure to identify a "significant" effect. You have to focus much more on statistical importance, such as effect size, rather than significance."

--- whuber

I was wondering if this is something that can be proved or simply some common phenomena in practice?

Any pointer to a proof/discussion/simulation would be really helpful.

• Effect size matters. (+1 to Glen_b's answer). To give a quick example: if we were obese we would not change our existing diet to a new more expensive diet if it resulted in weight-loss of 0.05 kg after a month even if it had a $p$-value $\leq 0.0000000001$. We would still be obese, just poorer. For all we know such a minor weight decrease might be just due to the health-clinic that the recordings where taken moving from the ground of a building with no elevator to the fourth floor of the same building. (Nice question + 1) Sep 3, 2017 at 13:36

It's pretty much general.

Imagine there's a small, but non-zero effect (i.e. some deviation from the null that the test is able to pick up).

AT small sample sizes, the chance of rejecting will be very close to the type I error rate (noise dominates the small effect).

As sample sizes grow the estimated effect should converge to that population effect, while at the same time the uncertainty of the estimated effect shrinks (normally as $\sqrt{n}$), until the chance that the null situation is close enough to the estimated effect that it's still plausible in a randomly selected sample from the population reduces to effectively zero.

Which is to say, with point nulls, eventually rejection becomes certain, because in almost all real situations there's essentially always going to be some amount of deviation from the null.

• "...because in almost all real situations there's essentially always going to be some amount of deviation from the null." So it's there and one can even see it. That would be a rather nice property or wouldn't it? Sep 3, 2017 at 12:59
• "Null" here refers to the null hypothesis that the coefficient is equal to zero? Sep 3, 2017 at 17:18
• I think Glen_b's answer is general and applicable to any hypothesis testing with a point null. In the context of regression, yes, the null is that the coefficient equals zero. My own understanding though... Sep 3, 2017 at 22:11

This is not a proof, but it is not hard to show the influence of the sample size in practice. I would like to use a simple example from Wilcox (2009) with minor changes:

Imagine that for a general measure of anxiety, a researcher claims that the population of college students has a mean of at least 50. As a check on this claim, suppose that ten college students are randomly sampled with the goal of testing $H_0: \mu \geq 50$ with $\alpha = .05$. (Wilcox, 2009: 143)

We can use t-test for this analysis:

$$T = \frac{\bar X - \mu_o}{s/\sqrt{n}}$$

Assuming that sample mean ($\bar X$) is 45 and sample standard deviation ($s$) is 11,

$$T = \frac{45-50}{11/\sqrt{10}}=-1.44.$$

If you look at a table containing critical values of Student's $t$ distribution with $ν$ degrees of freedom, you will see that the for $v = 10 -1$, $P(T \leq - 1.83)= .05$. So with $T=-1.44$, we fail to reject the null hypothesis. Now, let's assume we have same sample mean and standard deviation, but 100 observations instead:

$$T = \frac{45-50}{11/\sqrt{100}}= -4.55$$

For $v = 100 - 1$, $P(T \leq -1.66) = .05$ , we can reject the null hypothesis. Keeping everything else constant, increasing the sample size will decrease the denominator and you will more likely to have values in the critical (rejection) region of the sampling distribution. Note that $s/\sqrt{n}$ is an estimate of the standard error of the mean. So, you can see how a similar interpretation applies to, for example, the hypothesis tests on the regression coefficients obtained in linear regression, where $T = \frac{\hat\beta_j-\beta_j^{(0)}}{se(\hat\beta_j)}$.

Wilcox, R.R., 2009. Basic Statistics: Understanding Conventional Methods and Modern Insights. Oxford University Press, Oxford.

• Thanks for the answer. Your answer provides a concrete demo of Glen_b's answer: when the sample size is very large, tiny deviation from the null (there is always tiny deviation in practice) will be captured as significant effect. Sep 3, 2017 at 4:04

In regression, for the overall model, the test is on F. Here

$$F = \frac{\frac{RSS_1-RSS_2}{p_2 - p_1}}{\frac{RSS_2}{n-p_2}}$$ Where RSS is residual sum of squares and p is the number of parameters. But, for this question, the key is the N in the lower denominator. No matter how close $RSS_1$ is to $RSS_2$, when N gets bigger, F gets bigger. So, just increase N until F is significant.

• Thanks for the answer. However, I am skeptical about "when N gets bigger, F gets bigger"; when N increases, RSS2 increases as well, it is not clear to me why F will get bigger. Sep 3, 2017 at 3:50
• @Peter Flom this is unrealted but can you take a look here stats.stackexchange.com/questions/343518/… Apr 30, 2018 at 1:02