# Are the precision gains from stratified sampling a free lunch?

I'm following the discussion in Field Experiments by Gerber and Green, Chapter 3 as well as these resources:

Gerber and Green claim that under reasonable assumptions, blocking improves precision:

By making a small design change, we greatly improve the precision with which we estimate the [Average Treatment Effect]. (Ch. 3)

I wrote an R script to explore this:

total_sample <- 1000

# Overall mean under stratification
strat_mean <- function(sample_sizes, props){
return(sum(sample_sizes*props)/sum(sample_sizes))
}

# Overall SE under stratification
strat_se <- function(sample_sizes, props){
s <- sample_sizes / sum(sample_sizes)
s_sq <- s**2
se <- props*(1-props) / sum(sample_sizes)
return(sqrt(sum(s_sq*se)))
}

# Overall mean under stratification
normal_mean <- function(sample_sizes, props){
return(sum(sample_sizes*props)/sum(sample_sizes))
}

# Overall SE without stratification
normal_se <- function(sample_sizes, props){
p <- sum(sample_sizes*props)/sum(sample_sizes)
# p(1-p)/n
return(sqrt(p*(1-p)/sum(sample_sizes)))
}

# Two stratified blocks, each half of the sample
strat_props <- c(0.5, 0.5)
strat_ns <- strat_props*total_sample

# Observed proportion of 0.05 in both groups
observed_props <- c(0.05, 0.05)

print("stratified")
print(strat_mean(strat_ns,observed_props))
print(strat_se(strat_ns,observed_props))

print("normal")
print(normal_mean(strat_ns,observed_props))
print(normal_se(strat_ns,observed_props))


Which returns

[1] "stratified"
[1] 0.05
[1] 0.004873
[1] "normal"
[1] 0.05
[1] 0.006892


In this script, the stratification adds no information (we split the sample 50/50, and the proportion outcome we are measuring is the same between both groups).

However, the standard error for the proportion is lower under stratified sampling. This seems like a free lunch! If this is true, it seems to me that researchers could generate completely random "blocks" for their study and show radically improved "precision" without adding any additional information to the study.

How can this be the case?

• Could you add an explanation of what the R code does and the results – Juho Kokkala May 12 '18 at 19:05
• @JuhoKokkala I added comments to the code and provided the output. Let me know if that's not clear. – TuringMachin May 13 '18 at 14:13
• I think this may be a duplicate of this question where I showed that in non-trivial cases stratification nets you a decreased variance. For those who know more than I do about this, please feel free to prove me wrong, but the formulations and logic that I followed come from the referenced text. – Chris C May 22 '18 at 19:41
• Larry (accepted answer) found the issue. There's a bug in the code driving down the SEs. Fixing the bug yields the same SEs for both stratified and non-stratified method in this example. – TuringMachin May 22 '18 at 20:47

I believe your formula for the stratified SE is incorrect -

se <- props*(1-props) / sum(sample_sizes)

should be

se <- props*(1-props) / sample_sizes

Since se is a vector where the first component is the estimated se for the first strata and the second arg is the estimated se for the second strata. Your code as written is creating a vector where each strata is size 1000, not 500, so your precision gains are exactly the same as if you had doubled the overall sample size.

The precision comes from explaining explainable variation in the outcome variable. You can get the same precision gain by not stratifying in the sampling but doing covariate adjustment on the factors you would have stratified on. Stratification just accomplishes limiting the sample size in certain strata to concentrate resources elsewhere.

• I believe the example I included demonstrates that we gained precision without exploiting explainable variation in the outcome variable. Hence my question. LMK if I'm missing something. – TuringMachin May 14 '18 at 15:43
• If the stratifying variables are unrelated to $Y$ I don't see how you can gain any precision by stratification. – Frank Harrell May 15 '18 at 13:57
• Agreed! And yet somehow I get increased precision when I follow the formulas in the text. Hence the question. Am I misinterpreting the formulas? Is there something else going on? – TuringMachin May 15 '18 at 19:13