How large a study would you need to deal with the 10X programmer? The theory of the 10X programmer states that there is a range of 10X between the fastest and slowest programmer with a given amount of experience. Or in other words, if the fastest programmer takes 10 minutes, the slowest programmer in the group takes 100 minutes to perform the same task.
I have two groups, lets say group A uses Test Driven Development and group B, the control group, doesn't.
I find that on average group A is 30% faster than group B, with the usual 10X range in both groups. How many people do I need in each group for this finding to be statistically meaningful?
 A: We need to make some reasonable assumptions to get an impression of the numbers: Uniformely distributed programmers are one and we compute with the 10 to 100 example in the slower group.
Standard deviation of a large group of programmers uniformely in between 10 and 100 minutes mixed with one 1.0-30%=0.7 times this time:
sd(c(runif(1e6, 10, 100), .7*runif(1e6,10,100)))
is roughly 24. For the effect size, we need the difference in means.
55 - .7*55 = .3*55 = 16.5
library(pwr)
power.t.test(d = 16.5/55, power=.9)

results in
Two-sample t test power calculation 

n = 234.4628
delta = 0.3
sd = 1
sig.level = 0.05
power = 0.9
alternative = two.sided

NOTE: n is number in *each* group

Of course, some people do all the work for a power of just 80%. Of course, if we assume normally distributed programmers, we might find a smaller standard deviation. For all practical purposes, you'll have to make some assumptions which are plausible but not shure. Therefore power calculation in the real world often gives but a hint towards the appropriate numbers.
