How to calculate sample size for this study design?

I am trying to find out whether the number of participants I recruited (N=42) so far is sufficient for my study in which: there are two interfaces (say A and B). ALL students in the study will try the interface, A, in the first week and then the interface, B, in the second week. The results from both attempts will be rated by experts in 1-5 scale (where 1 being very bad, 3 being neutral and 5 being very good). Alpha is 0.05. And effect size 0.5 (Cohen). The null hypothesis is that there is no difference between the outcomes (as rated by experts) between interface A vs. B. I visited online power/sample size calculators like this and this, but I don't know the "mu" of the population nor am I sure that my design is one-tailed or two-tailed (although I'd guess it's one-tailed? Correct me if I'm wrong).

I'm not very familiar with statistics, so I need help from more knowledgeable people like the ones from this community. Thank you for your help in advance.

• @Penguin_Knight, I meant Cohen. Sorry! Commented Nov 13, 2014 at 18:36

In your case, there are a couple possibilities:

As paired sample t-test

Set it to t test, Means: Difference from constant, Post hoc. The punch in the Effect size estimate, alpha and sample size. (You may click on "Determine" to find out how the effect sizes are calculated.) Power is 0.8856.

As Wilcoxon singed-rank

Following similar steps, change the analysis to Means: Wilcoxon signed-rank test (matched pairs), you can also estimate the power for Wilcoxon. It's about 0.8703.

The host site also has documentation and tutorials. Feel free to read up if you have other new scenarios.

For t-test, it's more conservative to go for 2-tailed because your mean delta can be bigger or smaller than equality (aka 0).

• thank you so much! This is exactly what I needed. I'll download G Power and try. One related question: If I split participants into four groups of ten people each (AA, BB, AB, BA), then could I still use the same configurations you used above such as paired-sample two-tail t-test (especially for AA and BB) to calculate the power? Thanks so much again. Commented Nov 13, 2014 at 20:04
• @user1330974, if you randomize the sequence AB and BA, then there is no difference; in fact it's encouraged. However, if you mix in AA and BB, then i) it's a completely different research question and ii) the effect size cannot be reasonably the same between homogeneous pairs and heterogeneous pair. If you want to check reliability of the raters, then formally set up some reliability test. Don't waste 50% of your sample for quality check. Commented Nov 14, 2014 at 13:45
• Thank you very much for the great insights. :) I figured that mixing AA and BB into the experiment will make it even more messy. Assuming that I test AA and BB (no more testing AB and BA) and assuming if I only have 10 people for each group, is there a way to calculate power for that and if so, could you please enlighten me on the configurations I should choose in G-Power? (I started using it yesterday. THANKS very much for letting me know of its existence!) :) Commented Nov 14, 2014 at 18:47
• thanks for your answer. I have one question concerning the Effect size dz in your above snapshot. You set it as 0.5. But if I enter the mean and SD values in the effect size calculator in g*power, the effect size is often above 1. Can I set the effect size as yours? and is it according to Cohen's medium level of effect size? Commented Dec 22, 2020 at 7:07
• @NingyuePeng, first you'll need to click the "Determine =>" button to open the calculator panel, then enter the mean and SD, click "Calculate and transfer to the main window," and it should be entered into the effect size. And yes, it's based on Cohen's d so you can directly enter that as well. Commented Dec 22, 2020 at 18:03