When you combine questions you increase the reliability of the measure, and with a more reliable measure, you have more power.
The increase in reliability that you get from more questions is estimated using the Spearman Brown prophecy formula, here's the Wikipedia page: https://en.wikipedia.org/wiki/Spearman%E2%80%93Brown_prediction_formula (Oh, it calls it the prediction formula; same thing).
Here's how to think about this. The variance of the sum of two uncorrelated variables is equal to the sum of their variances. The error of each question is random (by definition), so the error standard deviation increases with the square root of the number of items. But the effect size is the same - it's correlated 1, so the difference between the groups increases linearly with the number of items. The difference between the means is getting larger faster than the standard deviation is getting larger, so the effect size is increasing.
You can also think of this in terms of information. I have a group of people, and I ask them some questions. I can increase my power (and my probability of a significant result) by increasing the number of questions that I ask.
Here's a practical example. I have two groups, and an outcome which has (population) sd=1. The difference between the two groups is 0.1. In a sample of 100, I have very little power to detect that effect.
Check R, and it appears my power is about 0.10.
> power.t.test(d=0.1, n=100)
Two-sample t test power calculation
n = 100
delta = 0.1
sd = 1
sig.level = 0.05
power = 0.104507
alternative = two.sided
NOTE: n is number in *each* group
But what if I had 100 questions, all with the same effect size? And I will sum these items. The variance will be equal to the variance (which is 1) times 100 = 100, so the standard deviation will be 10.
But what of the difference? We just add that up - so the expected difference between the groups is 100 - 0.1, which is equal to 10. Now our expected difference is equal to the standard deviation. Plug that into a power calculation and our power is 0.99999.
> power.t.test(d=1, n=100)
Two-sample t test power calculation
n = 100
delta = 1
sd = 1
sig.level = 0.05
power = 0.9999998
alternative = two.sided
NOTE: n is number in *each* group
That's the theory, let's try it in practice:
set.seed <- 1234
#create empty data frame
df <- as.data.frame(matrix(rnorm(10000), nrow=100))
#add names
names(df) <-paste0("y", 1:100)
#create group variable
df$group <- c(rep(0, 50), rep(1, 50))
#Create difference score
es <- 0.1
df[,paste0("y", 1:100)] <- apply(df[,paste0("y", 1:100)], 2, function(x) x + es * df$group)
#do some t-tests
t.test(df$y1 ~ df$group)
t.test(df$y2 ~ df$group)
#We're going to get bored before we do them all. Let's just get the p-values.
pValues <- apply(df[,paste0("y", 1:100)], 2, function(x) t.test(x ~ df$group)$p.value)
#How many are sig?
sum(pValues < 0.05)
What do we get?
> sum(pValues < 0.05)
[1] 11
11 - very close to the 10 that we expected.
But now let's get the sum of the items:
#Calculate the total
df$total <- apply(df[,paste0("y", 1:100)], 1, sum)
And do a t-test:
#Now t-test on total
t.test(df$total ~ df$group)
Welch Two Sample t-test
data: df$total by df$group
t = -7.1753, df = 95.758, p-value = 1.527e-10
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-16.020834 -9.077376
sample estimates:
mean in group 0 mean in group 1
-1.418933 11.130173
Monstrously significant, as we expected, the difference is close to 10 (as we expected) and the confidence intervals include 10.
It's also possible to test whether the size of the effect is different for different items - I'm assuming it's always the same. But that's another question. :)
