Unbiased estimate for treatment effect in block randomized experiment where probability of treatment varies by block

Question

I want to analyze the results of a block randomized experiment. Within each block, units are randomized to treatment and control as if that block were a completely randomized experiment. Specifically, I want an unbiased estimator for the overall treatment effect and an estimate of the variance of this estimator.

If the probability of receiving treatment is the same in each block, then an unbiased estimate of the overall average treatment effect ATE can be obtained like so:

$$\hat{ATE} = \sum_{j=1}^J \frac{N_j}{N}\hat{ATE_j}$$ Where:

• N = Total Number of units in the experiment
• $N_j$ = Number of units in block j
• $\hat{ATE_j}$ is the Average Treatment Effect in block j. Specifically, $$ \hat{ATE_j} = \bar{Y}_{j,treatment} - \bar{Y}_{j,control}$$

$\bar{Y}_{j,treatment}$ is the average observed outcome in the treatment group in block j and $\bar{Y}_{j,control}$ is the average observed outcome in the control group in block j.

The difficulty is that the probability of treatment varies by the strata/blocks. So, some blocks have 50% of their units treated and others have, for example, 10% treated. I've been told that the above estimator for ATE is only unbiased if the probability of treatment is constant across the blocks and that, otherwise, I need to use some kind of Horvitz-Thompson weighting, weighting the units by their inverse probability of treatment, in order to get an unbiased estimator.

I can't find any literature on this. Do I need to use horvitz-thompson or some other kind of weighting? Or can I still use the estimator above? Any references would be especially appreciated.

Thanks!

Answer 1

I don't think this is the case. Each $ATE_j$ is unbiased, so weighted average of them should be unbiased as well because the expectation is linear operator.

Here's a 500 rep simulation in Stata where the treatment effect is set to 15 with 50 blocks of 100 observations (on average):

capture program drop mybs
program define mybs, rclass
clear
qui set obs 5
gen block = _n
gen p_treat = uniform()
gen n = rpoisson(100)
qui expand n
gen treated = uniform()<p_treat
gen y = rnormal(100,15)
qui replace y = y + 15 if treated == 1
collapse (mean) y (count) n=y, by(block treated)
qui reshape wide y n, i(block) j(treated)
qui gen TE = y1-y0
qui gen N=n0+n1
qui sum TE [fw=N]
return scalar ate = r(mean) 
end

set seed 123456
simulate ate = r(ate), reps(500): mybs
sum ate, detail

There does not appear to be any bias here:

. sum ate, detail

                           r(ate)
-------------------------------------------------------------
      Percentiles      Smallest
 1%     9.240335       7.301056
 5%     11.53554       7.853343
10%     12.34418       8.115672       Obs                 500
25%     13.81046       8.936302       Sum of Wgt.         500

50%     15.07377                      Mean           15.03785
                        Largest       Std. Dev.      2.191757
75%     16.31851       20.99205
90%     17.74527       21.72955       Variance         4.8038
95%     18.50914       21.98358       Skewness      -.0549813
99%     20.30977       24.24625       Kurtosis       4.139082