I am trying to find out whether the number of participants I recruited (N=42) so far is sufficient for my study in which: there are two interfaces (say A and B). ALL students in the study will try the interface, A, in the first week and then the interface, B, in the second week. The results from both attempts will be rated by experts in 1-5 scale (where 1 being very bad, 3 being neutral and 5 being very good). Alpha is 0.05. And effect size 0.5 (Cohen). The null hypothesis is that there is no difference between the outcomes (as rated by experts) between interface A vs. B. I visited online power/sample size calculators like this and this, but I don't know the "mu" of the population nor am I sure that my design is one-tailed or two-tailed (although I'd guess it's one-tailed? Correct me if I'm wrong).

I'm not very familiar with statistics, so I need help from more knowledgeable people like the ones from this community. Thank you for your help in advance.

  • $\begingroup$ @Penguin_Knight, I meant Cohen. Sorry! $\endgroup$ – user1330974 Nov 13 '14 at 18:36

Go download G*Power here and install.

In your case, there are a couple possibilities:

As paired sample t-test

enter image description here

Set it to t test, Means: Difference from constant, Post hoc. The punch in the Effect size estimate, alpha and sample size. (You may click on "Determine" to find out how the effect sizes are calculated.) Power is 0.8856.

As Wilcoxon singed-rank

Following similar steps, change the analysis to Means: Wilcoxon signed-rank test (matched pairs), you can also estimate the power for Wilcoxon. It's about 0.8703.

enter image description here

The host site also has documentation and tutorials. Feel free to read up if you have other new scenarios.

For t-test, it's more conservative to go for 2-tailed because your mean delta can be bigger or smaller than equality (aka 0).

  • $\begingroup$ thank you so much! This is exactly what I needed. I'll download G Power and try. One related question: If I split participants into four groups of ten people each (AA, BB, AB, BA), then could I still use the same configurations you used above such as paired-sample two-tail t-test (especially for AA and BB) to calculate the power? Thanks so much again. $\endgroup$ – user1330974 Nov 13 '14 at 20:04
  • $\begingroup$ @user1330974, if you randomize the sequence AB and BA, then there is no difference; in fact it's encouraged. However, if you mix in AA and BB, then i) it's a completely different research question and ii) the effect size cannot be reasonably the same between homogeneous pairs and heterogeneous pair. If you want to check reliability of the raters, then formally set up some reliability test. Don't waste 50% of your sample for quality check. $\endgroup$ – Penguin_Knight Nov 14 '14 at 13:45
  • $\begingroup$ Thank you very much for the great insights. :) I figured that mixing AA and BB into the experiment will make it even more messy. Assuming that I test AA and BB (no more testing AB and BA) and assuming if I only have 10 people for each group, is there a way to calculate power for that and if so, could you please enlighten me on the configurations I should choose in G-Power? (I started using it yesterday. THANKS very much for letting me know of its existence!) :) $\endgroup$ – user1330974 Nov 14 '14 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.