I am designing an experiment and want to define the sample size. To identify this, I am setting my significance level to 0.05 and the power to 0.8. My alternative hypotheses says the two means should be at least 15% different from each other. Does this 15% correspond to the effect size? I had a look at this and this, but I am still not sure how to interpret this 15% I am setting my alternative hypothesis in the context of power analysis.

I tried to calculate the sample size based on the assumption that the 15 % is the effect size:

from statsmodels.stats.power import TTestIndPower
# parameters for power analysis
effect = 0.15
alpha = 0.05
power = 0.9
# perform power analysis
analysis = TTestIndPower()
result = analysis.solve_power(effect, power=power, nobs1=None, ratio=1.0, alpha=alpha)
print('Sample Size: %.3f' % result)

The output is Sample Size: 934.954

This does not seem reasonable. I am not sure if I a doing it the right way.

Can someone help here?


I think you're referring to measuring effect size in Cohen's D. If so, the effect size would be the difference in means, divided by the pooled standard deviation.

So, let's say you're looking at groups' comprehension scores on a test. Let's say your control group scores an average of 50, standard deviation 10. And let's say your treatment group scores an average of 57.5, standard deviation 5.

Then the Cohen's D would be = (57.5-50) / ((10+5)/2) = 1

(Note that this assumes equal sized control and treatment groups. Deriving the pooled standard deviation is actually a bit more complicated than averaging the two groups' standard deviations, if the two groups have different sample sizes. See wikipedia page on Cohen's d, for example.)

In your code, 0.15 refers to the minimum effect size (in Cohen's D) that you want to be powered to detect. What this means for your data depends on your samples' standard deviation(s). You haven't collected the data yet, so you'll have to make assumptions here based on similar past experiments, the literature, your judgement, etc. Researchers often run multiple power calculations using different assumptions and create a table that shows the various sample sizes required given these different assumptions. For example, you could look at the sample sizes needed given various effect sizes, and/or given various standard deviation(s).

  • $\begingroup$ Does that mean the effete size should be measured based on Cohen's D? Then what does the 15% represent in this context? $\endgroup$ –  owise Sep 15 '18 at 13:30
  • $\begingroup$ Yes, you can measure effect size in Cohen's D. In my example, 15% was the difference between treatment (mean = 57.5) and control (mean = 50), where 7.5/50 = 15% difference between the groups. When doing a power calculation, you need to specify the groups' means in units AND the standard deviation(s) of the groups. Or you can use Cohen's D, where you don't have to specify the standard deviation(s). Does that make sense? $\endgroup$ – the_scheining Sep 15 '18 at 19:05
  • $\begingroup$ In your code, 0.15 almost definitely represents Cohen's D. What this 'means' for your data depends on your standard deviation(s). So if we slightly tweak my example of a comprehension test. For simplicity, let's say the standard deviation of both groups is 10. Cohen's D of 0.15 means an effect size (in correctly answered questions) of 1.5 questions. i.e. if your control group gets a score of 50 on average, your code is asking what sample size you would need to detect a treatment that raises their score by 1.5 points (where standard deviation is 10 for both groups). $\endgroup$ – the_scheining Sep 15 '18 at 19:11
  • $\begingroup$ It occurs to me that I've assumed you're talking about achieving a 15% difference in means between groups on some continuous variable. Please confirm that this is the case? The power calculation - and my comments - change if we're talking about a binary variable, i.e. 50% of Group A does a thing, whereas 57.5% of Group B does the thing. In that case you need to use a proportion test not a t-test. $\endgroup$ – the_scheining Sep 15 '18 at 19:31
  • $\begingroup$ It is a continuous variable. All what you said does make sense if we are measuring the power, but in this case I want to identify the sample size before conducting the experiment, that is why we don't can not calculate the standard deviation and thus we can measure the effect size in Cohen's D. That is what I could not understand. What would be the minimum sample size given that the difference between the means is expected to be 15% difference and the significance level is 0.05? $\endgroup$ –  owise Sep 15 '18 at 23:51

Different power analysis programs will allow different measures of effect size for different types of statistical analyses. "Means will very by 15%" is certainly an effect size, but it may not be usable by your software, in which case you might need to convert it to something else.

EDIT In your initial sentence you say you set power to 0.8 but your code has power at 0.9.

Also, you have effect = 0.15. What does that mean? You need to figure that out from the documentation for TTestIndPower. But it cannot be a % difference.

  • $\begingroup$ I added my python code to the question. What do you mean by it might not be useable by the software? $\endgroup$ –  owise Sep 15 '18 at 13:35
  • $\begingroup$ I mean that each type of power analysis software (R packages, SAS PROC POWER, G*Power, whatever) will want to have the input in a certain form. $\endgroup$ – Peter Flom - Reinstate Monica Sep 15 '18 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.