13
$\begingroup$

I was reading this question regarding large scale regression (link) where whuber pointed out an interesting point as follows:

"Almost any statistical test you run will be so powerful that it's almost sure to identify a "significant" effect. You have to focus much more on statistical importance, such as effect size, rather than significance."

--- whuber

I was wondering if this is something that can be proved or simply some common phenomena in practice?

Any pointer to a proof/discussion/simulation would be really helpful.

$\endgroup$
  • 1
    $\begingroup$ Effect size matters. (+1 to Glen_b's answer). To give a quick example: if we were obese we would not change our existing diet to a new more expensive diet if it resulted in weight-loss of 0.05 kg after a month even if it had a $p$-value $\leq 0.0000000001$. We would still be obese, just poorer. For all we know such a minor weight decrease might be just due to the health-clinic that the recordings where taken moving from the ground of a building with no elevator to the fourth floor of the same building. (Nice question + 1) $\endgroup$ – usεr11852 Sep 3 '17 at 13:36
10
$\begingroup$

It's pretty much general.

Imagine there's a small, but non-zero effect (i.e. some deviation from the null that the test is able to pick up).

AT small sample sizes, the chance of rejecting will be very close to the type I error rate (noise dominates the small effect).

As sample sizes grow the estimated effect should converge to that population effect, while at the same time the uncertainty of the estimated effect shrinks (normally as $\sqrt{n}$), until the chance that the null situation is close enough to the estimated effect that it's still plausible in a randomly selected sample from the population reduces to effectively zero.

Which is to say, with point nulls, eventually rejection becomes certain, because in almost all real situations there's essentially always going to be some amount of deviation from the null.

$\endgroup$
  • $\begingroup$ "...because in almost all real situations there's essentially always going to be some amount of deviation from the null." So it's there and one can even see it. That would be a rather nice property or wouldn't it? $\endgroup$ – Trilarion Sep 3 '17 at 12:59
  • $\begingroup$ "Null" here refers to the null hypothesis that the coefficient is equal to zero? $\endgroup$ – Arash Howaida Sep 3 '17 at 17:18
  • $\begingroup$ I think Glen_b's answer is general and applicable to any hypothesis testing with a point null. In the context of regression, yes, the null is that the coefficient equals zero. My own understanding though... $\endgroup$ – Bayesric Sep 3 '17 at 22:11
4
$\begingroup$

This is not a proof, but it is not hard to show the influence of the sample size in practice. I would like to use a simple example from Wilcox (2009) with minor changes:

Imagine that for a general measure of anxiety, a researcher claims that the population of college students has a mean of at least 50. As a check on this claim, suppose that ten college students are randomly sampled with the goal of testing $H_0: \mu \geq 50$ with $\alpha = .05$. (Wilcox, 2009: 143)

We can use t-test for this analysis:

$$T = \frac{\bar X - \mu_o}{s/\sqrt{n}}$$

Assuming that sample mean ($\bar X$) is 45 and sample standard deviation ($s$) is 11,

$$T = \frac{45-50}{11/\sqrt{10}}=-1.44.$$

If you look at a table containing critical values of Student's $t$ distribution with $ν$ degrees of freedom, you will see that the for $v = 10 -1$, $P(T \leq - 1.83)= .05$. So with $T=-1.44$, we fail to reject the null hypothesis. Now, let's assume we have same sample mean and standard deviation, but 100 observations instead:

$$T = \frac{45-50}{11/\sqrt{100}}= -4.55$$

For $v = 100 - 1$, $P(T \leq -1.66) = .05$ , we can reject the null hypothesis. Keeping everything else constant, increasing the sample size will decrease the denominator and you will more likely to have values in the critical (rejection) region of the sampling distribution. Note that $s/\sqrt{n}$ is an estimate of the standard error of the mean. So, you can see how a similar interpretation applies to, for example, the hypothesis tests on the regression coefficients obtained in linear regression, where $T = \frac{\hat\beta_j-\beta_j^{(0)}}{se(\hat\beta_j)}$.


Wilcox, R.R., 2009. Basic Statistics: Understanding Conventional Methods and Modern Insights. Oxford University Press, Oxford.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the answer. Your answer provides a concrete demo of Glen_b's answer: when the sample size is very large, tiny deviation from the null (there is always tiny deviation in practice) will be captured as significant effect. $\endgroup$ – Bayesric Sep 3 '17 at 4:04
2
$\begingroup$

In regression, for the overall model, the test is on F. Here

$$ F = \frac{\frac{RSS_1-RSS_2}{p_2 - p_1}}{\frac{RSS_2}{n-p_2}} $$ Where RSS is residual sum of squares and p is the number of parameters. But, for this question, the key is the N in the lower denominator. No matter how close $RSS_1$ is to $RSS_2$, when N gets bigger, F gets bigger. So, just increase N until F is significant.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the answer. However, I am skeptical about "when N gets bigger, F gets bigger"; when N increases, RSS2 increases as well, it is not clear to me why F will get bigger. $\endgroup$ – Bayesric Sep 3 '17 at 3:50
  • $\begingroup$ @Peter Flom this is unrealted but can you take a look here stats.stackexchange.com/questions/343518/… $\endgroup$ – user3022875 Apr 30 '18 at 1:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.