This is a question to hopefully untangle the sometimes confounded notions of statistical significance and effect size. As far as I understand, sample size intermediates the relationship between them, such that, while often low $p$-values will correspond to large effects, you can get significance for small effects with a large sample size, or vice-versa – an effect that you know is large may not reach significance because of an unsuitably small sample size.

Would it be correct to say that, given an independent samples t-test for a fixed sample size $N$ (2 groups of $N/2$ subjects each), the measure of statistical reliability ($p$) is really the same as (i.e. has a simple dependency on) the measure of effect size (e.g. Cohen's $d$)? In other words, if keeping $N$ constant, then significance ($p$-value) will be reached as soon as the effect size ($d$) exceeds a certain threshold?

  • 1
    $\begingroup$ Could you explain in what way the p-value is a "measure of statistical reliability"? I don't see a sense in which it could be interpreted that way. $\endgroup$ – Glen_b -Reinstate Monica Mar 16 '14 at 21:23

In Cohen's work, the definition of $d$ contains population quantities ($\sigma$), so you don't actually know $d$ for a t-test - each sample can only estimate $d$ (it's defined prospectively rather than after you have the sample, because it's for prospective power analysis).

If you redefine $d$ in sample terms -- to be $d^*=\frac{\bar x_1-\bar x_2}{s_p}$, then at fixed $(n_1,n_2)$, $d$ is just a scaled $t$ statistic, and so of course is related to the p-value via scaling and the inverse-cdf of the null distribution of the $t$.

In that sense, yes, you could define your rejection rule directly in terms of that sample-based $d^*$ and the value of $p$ would be monotonic-decreasing in that sample version of $d$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.