How is the true effect size represented in an explanatory figure for power?

Question

I am reading Sham & Purcell (2014). I need a clarification on how true effect size is represented in their box 1:

... while the investigator sets the probability of type 1 error ($\alpha$) to a desired level, the probability of type 2 error ($\beta$) and therefore statistical power are subject to factors outside the investigator’s control, such as the true effect size, and the accuracy and completeness of the data. ...

In particular, my idea is that the true effect size in the corresponding image is the distance between the means of the two distributions (or is it the actual frequency of the two distributions?). Both can affect the error rates. Can the true effect size be identified in such image?

• P. C. Sham and S. M. Purcell, “Statistical power and significance testing in large-scale genetic studies.” Nature Publishing Group, vol. 15, no. 5, pp. 335–346, May 2014.

Answer 1

Figures like the one in the paper are very common for explaining power, but I don't like them because you have both distributions in the same panel. That implies they could both exist at the same time (which they can't), and makes it confusingly similar to figures that show the distributions from two different groups, experimental and control, whereas these distributions do not refer to groups. I prefer to put the two distributions in different panels:

How the universe actually works is either represented by the top panel or the bottom panel (although you will never truly know which). But certainly they can't both exist. Moreover, these are not the distributions of your two groups. Instead, the top panel refers to the sampling distribution of the test statistic when the null is true. The bottom panel refers to the sampling distribution of the test statistic when the null is false and a specific alternative is true. Of note, the effect size is not represented anywhere in this figure (these are theorized distributions of test statistics, not data).

Now consider this figure:

We will consider these to be actual samples of weights of some women (gray) and men (yellow) plotted as histograms with kernel density lines overlaid and the group means represented by vertical dashed lines. Then, the distance between the means is the realized value of the (raw) effect size. The frequencies of men and women in the sample does affect power (see here), but is unrelated to the effect size. If we made the wildly unrealistic assumption that the sample means in this figure were exactly correct, the distance between the vertical lines would be the true (raw) effect size. (Note that effect sizes are often presented in standardized form, that could be done and displayed here by just having standardized the data first, but it doesn't happen to be what is in this figure.)

Answer 2

What you see in this image is the distribution of the sample mean under the null hypothesis and also the distribution of the sample mean under one particular example of the alternative hypothesis.

This particular example is chosen by the researcher as the minimum effect size that he would like to have enough power to detect. To take a silly example, if you are testing whether men are on average taller than women, you might say that you are only interested in size differences of 2cm or more. If you have a good idea of the population variance, you can estimate the distribution of the sample mean of the size difference for 2cm supposed size difference and different values of $n$. You select an $n$ that gives you adequate power to detect the minimum relevant effect size of 2cm.

The true effect size can only be estimated once the data is in. Power analysis should happen before the data is in. Therefore you cannot read the true effect size from your power analysis.