What happened to statistical significance in regression when data size is gigantic?

Question

I was reading this question regarding large scale regression (link) where whuber pointed out an interesting point as follows:

"Almost any statistical test you run will be so powerful that it's almost sure to identify a "significant" effect. You have to focus much more on statistical importance, such as effect size, rather than significance."

--- whuber

I was wondering if this is something that can be proved or simply some common phenomena in practice?

Any pointer to a proof/discussion/simulation would be really helpful.

Answer 1

It's pretty much general.

Imagine there's a small, but non-zero effect (i.e. some deviation from the null that the test is able to pick up).

AT small sample sizes, the chance of rejecting will be very close to the type I error rate (noise dominates the small effect).

As sample sizes grow the estimated effect should converge to that population effect, while at the same time the uncertainty of the estimated effect shrinks (normally as $\sqrt{n}$), until the chance that the null situation is close enough to the estimated effect that it's still plausible in a randomly selected sample from the population reduces to effectively zero.

Which is to say, with point nulls, eventually rejection becomes certain, because in almost all real situations there's essentially always going to be some amount of deviation from the null.

Answer 2

This is not a proof, but it is not hard to show the influence of the sample size in practice. I would like to use a simple example from Wilcox (2009) with minor changes:

Imagine that for a general measure of anxiety, a researcher claims that the population of college students has a mean of at least 50. As a check on this claim, suppose that ten college students are randomly sampled with the goal of testing $H_0: \mu \geq 50$ with $\alpha = .05$. (Wilcox, 2009: 143)

We can use t-test for this analysis:

$$T = \frac{\bar X - \mu_o}{s/\sqrt{n}}$$

Assuming that sample mean ($\bar X$) is 45 and sample standard deviation ($s$) is 11,

$$T = \frac{45-50}{11/\sqrt{10}}=-1.44.$$

If you look at a table containing critical values of Student's $t$ distribution with $ν$ degrees of freedom, you will see that the for $v = 10 -1$, $P(T \leq - 1.83)= .05$. So with $T=-1.44$, we fail to reject the null hypothesis. Now, let's assume we have same sample mean and standard deviation, but 100 observations instead:

$$T = \frac{45-50}{11/\sqrt{100}}= -4.55$$

For $v = 100 - 1$, $P(T \leq -1.66) = .05$ , we can reject the null hypothesis. Keeping everything else constant, increasing the sample size will decrease the denominator and you will more likely to have values in the critical (rejection) region of the sampling distribution. Note that $s/\sqrt{n}$ is an estimate of the standard error of the mean. So, you can see how a similar interpretation applies to, for example, the hypothesis tests on the regression coefficients obtained in linear regression, where $T = \frac{\hat\beta_j-\beta_j^{(0)}}{se(\hat\beta_j)}$.

---

Wilcox, R.R., 2009. Basic Statistics: Understanding Conventional Methods and Modern Insights. Oxford University Press, Oxford.

Answer 3

In regression, for the overall model, the test is on F. Here

$$ F = \frac{\frac{RSS_1-RSS_2}{p_2 - p_1}}{\frac{RSS_2}{n-p_2}} $$ Where RSS is residual sum of squares and p is the number of parameters. But, for this question, the key is the N in the lower denominator. No matter how close $RSS_1$ is to $RSS_2$, when N gets bigger, F gets bigger. So, just increase N until F is significant.