Probably a weird place to quote, but I just stumbled across this discussion on Reddit, and one of them said (with typos fixed):

The effect size in a power calculation has no relationship whatsoever to the minimal effect size under a non-point null. These are two completely unrelated. If you consider an effect size in your power calculation against a point null, you're still testing a point null.

(......) Of course you can have a null of zero in a power simulation. You usually do. You compute power at the alternative, but the test statistic is computed under the assumption of the null. You shouldn't be running around changing your null.

(......) You can use composite nulls, but this is in general very uncommon. Using a composite null is also different than running a power calculation for minimal effect size.

As indicated, the most obvious difference between (A) using a composite null and (B) running a power calculation for minimal effect size is that the sample distributions of statistics are calculated under the assumption of their respective null hypothesis. However, I'm not sure if it's the only difference. More importantly, what's their implications, and when should we use (A), (B), or both?

To put it more concretely, given that I specify the expected minimum effect size as "the actual difference in means is at least $0.8\sigma$", what the difference among the following configurations?

$$ \begin{align} H_0&: \mu = 0 & H_1&: \mu \ne 0 \tag 1\\ H_0&: -0.8\sigma \le \mu \le 0.8\sigma & H_1&: \mu \ne 0 \tag 2\\ H_0&: \mu = 0 & H_1&: \mu < -0.8\sigma \lor \mu > 0.8\sigma \tag 3 \end{align} $$

By the way, I'm not familiar with the terms "power analysis" and "effect size", so please don't assume me a strong background. Believe it or not, my professor didn't even mention them when teaching us hypothesis testing. (Yeah, as a Statistics major, this really sucks.)

  • 1
    $\begingroup$ Neither term was used in my stats major either, indeed not even in my PhD, but it's not because anything was lacking from my education; you have probably encountered all the necessary concepts, just not framed in this exact way. Effect size is usually some standardized measure of how false the null is (like $\frac{\mu_2-\mu_1}{\sigma}$ in a two sample test of means) and what people usually mean when they say "power analysis" is not an analysis of power at all; it's usually instead a calculation of the required sample size to achieve a particular power given an effect size of interest ... ctd $\endgroup$
    – Glen_b
    Jul 23 '19 at 5:14
  • $\begingroup$ ctd ... - e.g. we want at least an 80% chance to reject the null when the actual difference in means is (say) 0.4 standard deviations, what is the required sample size? ... why they call that power analysis instead of 'sample size calculation' is beyond me; to me a power analysis would be something along the lines of finding a power curve (typically as a function of effect size given a sample size, or as a function of sample size, given an effect size.) $\endgroup$
    – Glen_b
    Jul 23 '19 at 5:15
  • $\begingroup$ @Glen_b Can you elaborate on the relationship of "effect size", "null hypothesis" and "alternative hypothesis"? Your description suggests they are unrelated, and that the effect size is essentially a parameter of the test, like $\alpha$ and $\beta$. However, this paper defines the effect size as the distance between the null and the alternative: (...) These two groups differ by some degree: the effect size. $\endgroup$
    – nalzok
    Jul 23 '19 at 6:11
  • 1
    $\begingroup$ Not in comments. I was attempting to reassure you that your stats education was probably not lacking, it's a matter of what application areas focus on. Don't rely on any one source. Read widely and you'll find all manner of definitions of effect size. I suggest you stick with Jacob Cohen, though (where much of this usage of terminology and focus originates). In the subject areas that use these terms the alternatives are never simple alternatives, so "the alternative" doesn't really make sense, except in so far as they focus on a particular effect size. $\endgroup$
    – Glen_b
    Jul 23 '19 at 6:21
  • $\begingroup$ @Glen_b Thanks, will check out A power primer by Jacob Cohen! $\endgroup$
    – nalzok
    Jul 23 '19 at 6:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.