# A-priori power analysis for usability study of software

I want to conduct a usability study for a software programming tool. There are two versions of the tool (prototype A and prototype B) with different features, but allowing the same goals. I want the participants to complete a set of tasks using both versions of the tool and report the time taken. I'll compare the time taken by the two methods. The same set of participants perform tasks using prototype A and then prototype B. How can I find out the number of participants required for the study?

-

I would recommend the G*Power package which give multiple options to perform power calculations.

In your case it is likely that you use a paired t-test. So therefore you have to specify the following:

Test family = t tests
Statistical test = Means: Difference between two dependent means (matched pairs)
Type of power analysis = A priori: Compute required sample size - given a, power, and effect size
Input parameters = Tail(s) --> one/two (depends on what your hypothesis is)
Effect size dz --> (also depends on your hypothesis, use determine)
a err prop --> most logical is .05
Power --> specify power which you find expectable


Hit calculate and you get the required sample size (34 in case of two tailed, an effect size of .5, an alpha of .05, and a power of.80)

-
thanks.. it's hard before hand to determine the mean difference of times taken for the hypothesis. for significance testing if my null hypothesis is the mean times taken by the participants for prototype A is same as prototype B, then what values do i enter for calculating the effect size? – iceman Feb 27 '13 at 17:07
@iceman You'll need i) the suggested difference, ii) the SD of group A, and ii) the SD of group B. If you're not sure how to compute it, I believe in G*Power you can click some button next to the effect size entry field to expand a little calculator at the right hand side. There you can put in the statistics, and then get the effect size. – Penguin_Knight Feb 27 '13 at 17:25
@Penguin_Knight; thanks..yep, i found the "Determine=>" box to get the effect size calculator. it has two ways - "From Differences" and "From Group Parameters"...usually for means, the values come from literature or previous studies..i don't have that yet..so do i have to formulate it with a random value first..for example, the null hypothesis will be .."the mean difference in the time taken to complete the tasks is zero seconds" and the alternative hypothesis is that the mean difference is 100 seconds... – iceman Feb 27 '13 at 17:36
@iceman If you have nothing then an educated guess can serve as a good start. You can also create a few different scenarios for a range of sample sizes. – Penguin_Knight Feb 27 '13 at 17:39
Yes to @Penguin_Knight (and iceman!), a typical way under uncertainty like this is to graph the power of tests under various expected differences. Imagine a graph with sample size on the X axis and power on the Y-axis, you make plot several different lines for expected mean differences. If you are truly ignorant of the differences and can't even make reasonable guesses, a pilot study may be in order. – Andy W Feb 28 '13 at 12:31