Are the precision gains from stratified sampling a free lunch?

Question

I'm following the discussion in Field Experiments by Gerber and Green, Chapter 3 as well as these resources:

• http://ocw.jhsph.edu/courses/StatMethodsForSampleSurveys/PDFs/Lecture4.pdf
• http://home.iitk.ac.in/~shalab/sampling/chapter4-sampling-stratified-sampling.pdf

Gerber and Green claim that under reasonable assumptions, blocking improves precision:

By making a small design change, we greatly improve the precision with which we estimate the [Average Treatment Effect]. (Ch. 3)

I wrote an R script to explore this:

total_sample <- 1000

# Overall mean under stratification
strat_mean <- function(sample_sizes, props){
  return(sum(sample_sizes*props)/sum(sample_sizes))
}

# Overall SE under stratification    
strat_se <- function(sample_sizes, props){
  s <- sample_sizes / sum(sample_sizes)
  s_sq <- s**2
  se <- props*(1-props) / sum(sample_sizes)
  return(sqrt(sum(s_sq*se)))
}

# Overall mean under stratification
normal_mean <- function(sample_sizes, props){
  return(sum(sample_sizes*props)/sum(sample_sizes))
}

# Overall SE without stratification    
normal_se <- function(sample_sizes, props){
  p <- sum(sample_sizes*props)/sum(sample_sizes)
  # p(1-p)/n
  return(sqrt(p*(1-p)/sum(sample_sizes)))
}

# Two stratified blocks, each half of the sample
strat_props <- c(0.5, 0.5)
strat_ns <- strat_props*total_sample

# Observed proportion of 0.05 in both groups
observed_props <- c(0.05, 0.05)

print("stratified")
print(strat_mean(strat_ns,observed_props))
print(strat_se(strat_ns,observed_props))

print("normal")
print(normal_mean(strat_ns,observed_props))
print(normal_se(strat_ns,observed_props))

Which returns

[1] "stratified"
[1] 0.05
[1] 0.004873
[1] "normal"
[1] 0.05
[1] 0.006892

In this script, the stratification adds no information (we split the sample 50/50, and the proportion outcome we are measuring is the same between both groups).

However, the standard error for the proportion is lower under stratified sampling. This seems like a free lunch! If this is true, it seems to me that researchers could generate completely random "blocks" for their study and show radically improved "precision" without adding any additional information to the study.

How can this be the case?

Answer 1

I believe your formula for the stratified SE is incorrect -

se <- props*(1-props) / sum(sample_sizes)

should be

se <- props*(1-props) / sample_sizes

Since se is a vector where the first component is the estimated se for the first strata and the second arg is the estimated se for the second strata. Your code as written is creating a vector where each strata is size 1000, not 500, so your precision gains are exactly the same as if you had doubled the overall sample size.

Answer 2

The precision comes from explaining explainable variation in the outcome variable. You can get the same precision gain by not stratifying in the sampling but doing covariate adjustment on the factors you would have stratified on. Stratification just accomplishes limiting the sample size in certain strata to concentrate resources elsewhere.