I'm following the discussion in Field Experiments by Gerber and Green, Chapter 3 as well as these resources:
- http://ocw.jhsph.edu/courses/StatMethodsForSampleSurveys/PDFs/Lecture4.pdf
- http://home.iitk.ac.in/~shalab/sampling/chapter4-sampling-stratified-sampling.pdf
Gerber and Green claim that under reasonable assumptions, blocking improves precision:
By making a small design change, we greatly improve the precision with which we estimate the [Average Treatment Effect]. (Ch. 3)
I wrote an R script to explore this:
total_sample <- 1000
# Overall mean under stratification
strat_mean <- function(sample_sizes, props){
return(sum(sample_sizes*props)/sum(sample_sizes))
}
# Overall SE under stratification
strat_se <- function(sample_sizes, props){
s <- sample_sizes / sum(sample_sizes)
s_sq <- s**2
se <- props*(1-props) / sum(sample_sizes)
return(sqrt(sum(s_sq*se)))
}
# Overall mean under stratification
normal_mean <- function(sample_sizes, props){
return(sum(sample_sizes*props)/sum(sample_sizes))
}
# Overall SE without stratification
normal_se <- function(sample_sizes, props){
p <- sum(sample_sizes*props)/sum(sample_sizes)
# p(1-p)/n
return(sqrt(p*(1-p)/sum(sample_sizes)))
}
# Two stratified blocks, each half of the sample
strat_props <- c(0.5, 0.5)
strat_ns <- strat_props*total_sample
# Observed proportion of 0.05 in both groups
observed_props <- c(0.05, 0.05)
print("stratified")
print(strat_mean(strat_ns,observed_props))
print(strat_se(strat_ns,observed_props))
print("normal")
print(normal_mean(strat_ns,observed_props))
print(normal_se(strat_ns,observed_props))
Which returns
[1] "stratified"
[1] 0.05
[1] 0.004873
[1] "normal"
[1] 0.05
[1] 0.006892
In this script, the stratification adds no information (we split the sample 50/50, and the proportion outcome we are measuring is the same between both groups).
However, the standard error for the proportion is lower under stratified sampling. This seems like a free lunch! If this is true, it seems to me that researchers could generate completely random "blocks" for their study and show radically improved "precision" without adding any additional information to the study.
How can this be the case?