(Inverse) proportionality between significance and effect size, for a fixed sample size?

Question

This is a question to hopefully untangle the sometimes confounded notions of statistical significance and effect size. As far as I understand, sample size intermediates the relationship between them, such that, while often low $p$-values will correspond to large effects, you can get significance for small effects with a large sample size, or vice-versa – an effect that you know is large may not reach significance because of an unsuitably small sample size.

Would it be correct to say that, given an independent samples t-test for a fixed sample size $N$ (2 groups of $N/2$ subjects each), the measure of statistical reliability ($p$) is really the same as (i.e. has a simple dependency on) the measure of effect size (e.g. Cohen's $d$)? In other words, if keeping $N$ constant, then significance ($p$-value) will be reached as soon as the effect size ($d$) exceeds a certain threshold?

Answer 1

In Cohen's work, the definition of $d$ contains population quantities ($\sigma$), so you don't actually know $d$ for a t-test - each sample can only estimate $d$ (it's defined prospectively rather than after you have the sample, because it's for prospective power analysis).

If you redefine $d$ in sample terms -- to be $d^*=\frac{\bar x_1-\bar x_2}{s_p}$, then at fixed $(n_1,n_2)$, $d$ is just a scaled $t$ statistic, and so of course is related to the p-value via scaling and the inverse-cdf of the null distribution of the $t$.

In that sense, yes, you could define your rejection rule directly in terms of that sample-based $d^*$ and the value of $p$ would be monotonic-decreasing in that sample version of $d$.