1
$\begingroup$

When comparing two effect sizes with a statistical test, does it matter for the test's power whether these effect sizes themselves are large or small?

Imagine you are planning an experiment in which you compute an effect (difference between two conditions) for two groups (A,B). You want to show that the effect is larger in one group than the other, i.e. your hypothesis is effect(A) > effect(B). Assuming that the difference between the groups is equivalent to Cohen's D = 0.5, what is the required sample size to reach a power of 0.8?

In hypothetical experiment I the effect sizes in both groups are 1 and 1.5, the difference between them is 0.5.

In hypothetical expeirment II the effect sizes in both groups are 0.1 and 0.6. Again, the difference is 0.5.

Although the difference between the two groups is the same in both experiments, I would assume that experiment II requires a larger sample size. After all, the effects are located on a much smaller scale than in experiment 1. I have been told an a priori power analysis needs to take this into account (the magnitude of the effects that are being compared), but I do not really know how, and it's hard to find answers to this very specific problem on the web.

How can I take this into account in my a priori power analysis? I'm using G*Power and unfortunately, I can't simply answer this by simulating data (have to use a validated method).

$\endgroup$
  • $\begingroup$ Forgot to mention - for the sake of a priori power analysis, both groups would have the same variance and N; it's just their means that are different, and thus they show different effect sizes for a manipulation. $\endgroup$ – Chris Sep 10 '14 at 10:37
2
$\begingroup$

You seem to be mixing together these two things

  • the sample size needed to detect an effect of a certain size
  • the sample size needed to detect a difference of two effects being equal to a certain value

The difference between two effects is a different statistical animal than the effects themselves. So you need the same sample size in both cases to detect a difference of .5. But if your goals also include making inferences about the effects themselves, you need more data to detect smaller effects.

$\endgroup$
  • $\begingroup$ I will specifiy my case. We know that our control group will show a certain effect (quite a strong one, d = 0.88). We want to test whether another group shows the same effect, or a weaker one. We're interested in differences if they are relevant, which we defined as d = 0.5 (for the difference between two effects). You say: 'You need the same sample size in both cases to detect a difference of 0.5'. Does that mean that it does not matter how large the effects in both groups are, as long as they are 0.5 apart? $\endgroup$ – Chris Sep 10 '14 at 15:16
  • $\begingroup$ I think you're making things a bit too complicated by looking at the effect within groups as effects, by measuring them in e.g. Cohen's d. Think of them simply as values, whatever the unit may be. The effect you're interested in (for which you care about the d) is the difference between these values. $\endgroup$ – jona Sep 10 '14 at 15:42
  • $\begingroup$ "Does that mean that it does not matter how large the effects in both groups are, as long as they are 0.5 apart?" Yes, that's what I'm saying. I also agree with @jonas. I think an emphasis on standardized effect sizes obscures meaning, and people are better off talking about real differences of real measurements, in the units in which they are made. $\endgroup$ – Russ Lenth Sep 10 '14 at 19:45
  • $\begingroup$ Using real units isn't always that straightforward. In our case, the units from previous research are all kinds of biophysical measures as well as derived ones - so the only way to make these studies' results comparable was by looking at their test summary statistics (in t and F). And the reason that I went through the ordeal of collecting all that information was so I could construct hypothetical data for my power calculation. But if the effect strenghts in either group don't matter for testing the difference, that's fine by me. I'll verify ASAP and mark 'answered' or post on. Thanks already! $\endgroup$ – Chris Sep 11 '14 at 8:52
  • $\begingroup$ Verification was succesful, the level on which effects are located does not matter, only their difference :) $\endgroup$ – Chris Sep 30 '14 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.