2
$\begingroup$

Is it true that, given a fixed sample size N, it can be said that some measure of effect size and the measure of statitical significance (e.g., 1-p) are proportional? In other words, that, when keeping sample size constant, in order to reach significance you need to be trying to prove an effect that is large enough.

And is it correct to (simplistically) summarise the relationship between the three concepts (effect size, sample size, and statistical significance) with the following graph, which essentially describes the old adage "with large sample sizes, small effects become significant" graph summarising question

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ Very closely related: stats.stackexchange.com/questions/91561, stats.stackexchange.com/questions/85878, and stats.stackexchange.com/a/80960. Looking at the power curves in the answers may be helpful. $\endgroup$
    – whuber ♦
    Commented May 5, 2014 at 19:28
  • $\begingroup$ proportional? No; the relationship isn't even linear. However, I'd never call $1-p$ "significance", since that term already has a well established meaning that is different from that (you have significance when $p<\alpha$; it's a 0-1 step function in $p$, not a smooth thing like $1-p$). $\endgroup$
    – Glen_b
    Commented May 6, 2014 at 0:00

1 Answer 1

Reset to default
1
$\begingroup$

As an example, look at the relationship between t, p and d in the t-test.

p is defined as the probability of obtaining a test statistic this or more extreme, if the hypothesis under test is true. It is inversely related to both the sample size and the test statistic - in this case, t. Conversely, t ~ 1-p and n ~ 1-p.

For correlated samples, $d_{z} = \frac{t}{\sqrt{n}}$. Then, $d_{z}\sqrt{n} = t$; then, if n is held constant, as $d_{z}$ increases, t increases; and since t ~ 1-p, $d_{z}$ ~ 1-p. Also, if $d_{z}$ is held constant, as n increases, t increases, and n ~ 1-p. Colloquially speaking, since both 1-p and the measures of effect size depend on how extreme the test statistic is, they are correlated.

This will apply to other measures of effect size as well, such as the relationship between F, n and $\eta^2$ - trivially for those cases where the F-test converges with the t-test; or for correlation coefficients, which can also converge with the t-test. (I'd be very interested to learn if there are any measures of effect size which do not behave in this way, and by which justification they could be associated with other standardised measures of effect size.)

in order to reach significance you need to be trying to prove an effect that is large enough

Note that an effect size (such as r) is a descriptive statistic, a property of the sample. It is not inferential. To support a claim about the population, confidence intervals over effect sizes can be used.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.