Is it true that, given a fixed sample size N, it can be said that some measure of effect size and the measure of statitical significance (e.g., 1-p) are proportional? In other words, that, when keeping sample size constant, in order to reach significance you need to be trying to prove an effect that is large enough.

And is it correct to (simplistically) summarise the relationship between the three concepts (effect size, sample size, and statistical significance) with the following graph, which essentially describes the old adage "with large sample sizes, small effects become significant" graph summarising question


As an example, look at the relationship between t, p and d in the t-test.

p is defined as the probability of obtaining a test statistic this or more extreme, if the hypothesis under test is true. It is inversely related to both the sample size and the test statistic - in this case, t. Conversely, t ~ 1-p and n ~ 1-p.

For correlated samples, $d_{z} = \frac{t}{\sqrt{n}}$. Then, $d_{z}\sqrt{n} = t$; then, if n is held constant, as $d_{z}$ increases, t increases; and since t1-p, $d_{z}$ ~ 1-p. Also, if $d_{z}$ is held constant, as n increases, t increases, and n ~ 1-p. Colloquially speaking, since both 1-p and the measures of effect size depend on how extreme the test statistic is, they are correlated.

This will apply to other measures of effect size as well, such as the relationship between F, n and $\eta^2$ - trivially for those cases where the F-test converges with the t-test; or for correlation coefficients, which can also converge with the t-test. (I'd be very interested to learn if there are any measures of effect size which do not behave in this way, and by which justification they could be associated with other standardised measures of effect size.)

in order to reach significance you need to be trying to prove an effect that is large enough

Note that an effect size (such as r) is a descriptive statistic, a property of the sample. It is not inferential. To support a claim about the population, confidence intervals over effect sizes can be used.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.