# A “contradictory” derivation of the variance of two estimators

Consider a paired-experiment setting. I have a variable $$Z=\{1,2,...,N\}$$, based on which, I paired the subjects. For each value of $$Z$$, I have 2 subjects, and randomly assign one of them to the Treatment and the other to the Control. So it looks like this

Z  T  C
1  .  .
2  .  .
...
N  .  .


Let $$Y^T$$ be random variable denoting the metric of the response variable when Treatment is given and likewise for $$Y^C$$.

Now we could pretend that we didn't conduct the paired-experiment and we pool the data together (assuming we just collect $$2N$$ people and randomly assigned half of them to T, the rest to C and it happens that the variable $$Z$$ is balanced). Then we could use the estimator $$\mathcal{E}_1=\frac{1}{|T|}\sum_{i\in T}Y^T_i-\frac{1}{|C|}\sum_{i\in C}Y^C_i=\frac{1}{N}\left(\sum_{i=1}^NY^T_i-\sum_{i=1}^NY^C_i\right)$$ to estimate the population treatment effect. Assuming $$Y^T$$ and $$Y^C$$ are uncorrelated, then we have $$Var[\mathcal{E}_1]=\frac{\sigma^2_T}{N}+\frac{\sigma^2_C}{N},$$ where $$Var[Y^T]=\sigma^2_T$$ and $$Var[Y^C]=\sigma^2_C$$.

Now if we utilize the paired-experiment design, we could use another estimator $$\mathcal{E}_2=\sum_{k}\frac{|T_k|+|C_k|}{|T|+|C|}\bigg(\frac{1}{|T_k|}\sum_{i\in T_k}Y^A_i-\frac{1}{|C_k|}\sum_{j\in C_k}Y^B_j\bigg)=\frac{1}{N}\sum_{i=1}^N(Y^T_i-Y^C_i),$$ where $$k$$ is the value that $$Z$$ takes and in this example $$|T_k|=|C_k|=1,~\forall~k$$.

Because we assume $$Y^T$$ and $$Y^C$$ are uncorrelated, then we have $$Var[\mathcal{E}_2]=\frac{\sigma^2_T}{N}+\frac{\sigma^2_C}{N}=Var[\mathcal{E}_1]$$ So in this example, paired-experiment does not help us to get an estimator with lower variance. And both $$\mathcal{E}_1$$ and $$\mathcal{E}_2$$ are unbiased estimator of the population treatment effect and in this case, they are identical.

However, the following derivation shows that $$Var[\mathcal{E}_2] and I don't know where goes wrong \begin{align*} &Var(Y^T)=\sigma^2_T=\mathbb{E}[Var(Y^T|Z)]+Var(\mathbb{E}[Y^T|Z])\\ &Var(Y^C)=\sigma^2_C=\mathbb{E}[Var(Y^C|Z)]+Var(\mathbb{E}[Y^C|Z]) \end{align*} Then, let $$w_k=\frac{|T_k|}{|T|}=\frac{|C_k|}{|C|}=1/N$$ \begin{align*} Var(\mathcal{E}_2)&=\sum_{k}\bigg(\frac{|T_k|+|C_k|}{|T|+|C|}\bigg)^2Var\bigg(\frac{1}{|T_k|}\sum_{i\in T_k}Y^T_i-\frac{1}{|C_k|}\sum_{j\in C_k}Y^C_j\bigg)\\ &=\sum_{k}\bigg(\frac{|T_k|+|C_k|}{|T|+|C|}\bigg)^2\bigg(\frac{1}{|T_k|}Var(Y^T|Z=k)+\frac{1}{|C_k|}Var(Y^C|Z=k)\bigg)\\ &=\sum_{k}w_k^2\bigg(\frac{1}{w_k|T|}Var(Y^T|Z=k)+\frac{1}{w_k|C|}Var(Y^C|Z=k)\bigg)\\ &=\frac{1}{|T|}\sum_{k}w_kVar(Y^T|Z=k)+\frac{1}{|C|}\sum_{k}w_kVar(Y^C|Z=k)\\ &=\frac{1}{|T|}\mathbb{E}[Var(Y^T|Z)]+\frac{1}{|C|}\mathbb{E}[Var(Y^C|Z)]<\frac{\sigma^2_T}{|T|}+\frac{\sigma^2_C}{|C|}=Var(\mathcal{E}_1) \end{align*}

• In your "following derivation" you are, in effect, performing a stratified sampling analysis. If the strata ($Z$) and the outcomes ($Y^T$ and $Y^C$) are independent, then the $<$ in your last line will actually be an $=$ sign and there's no issue. If they aren't independent, the $<$ sign will be correct, but it's because you're taking into account the de-facto stratification in your last estimation but not in your first two - which is well-known to reduce variance, otherwise no-one would do it! – jbowman Mar 10 '18 at 15:45

You made a couple of mistakes, first, in your initial derivation of $Var(\mathcal{E}_2)$, and second, when passing from the next-to-last line to the last line of your final derivation.

As an aside, there is also a mistake in your last line, which I will get out of the way now. You have:

$$\frac{1}{|T|}\mathbb{E}[Var(Y^T|Z)]+\frac{1}{|C|}\mathbb{E}[Var(Y^C|Z)]<\frac{\sigma^2_T}{|T|}+\frac{\sigma^2_C}{|C|}=Var(\mathcal{E}_1)$$ but the way iterated expectations work means that:

$$\mathbb{E}[Var(Y^T|Z)] = Var(Y^T) \equiv \sigma^2_T$$

and similarly $\mathbb{E}[Var(Y^C|Z)] = Var(Y^C) \equiv \sigma^2_C$. Substituting shows that the inequality sign should actually be an equality sign.

On to the main problem! Typically when performing stratified sampling we do so with the intent of reducing variance. This comes about because the variance of observations of our test statistic are reduced by conditioning upon the stratification variable(s). In the case of your example, this translates to:

$$Var(Y^T|Z) \leq \sigma^2_T$$

and similarly for $Var(Y^C|Z)$. Now, going to the next-to-last line of your second derivation, we have:

$$\frac{1}{N|T|}\sum_{k}Var(Y^T|Z=k)+\frac{1}{N|C|}\sum_{k}Var(Y^C|Z=k)$$

If stratification buys us nothing, then $Var(Y^T|Z) = Var(Y^T) \equiv \sigma^2_T$ and similarly for $Var(Y^C|Z)$. Substituting gives us:

$$\frac{1}{N|T|}\sum_{k}\sigma^2_T+\frac{1}{N|C|}\sum_{k}\sigma^2_C = \frac{\sigma^2_T}{|T|}+\frac{\sigma^2_C}{|C|}$$

which is in agreement with your result for $Var(\mathcal{E}_2)$. This is what the last line of your derivation should have been.

If stratification is of benefit, then:

$$Var(Y^T-Y^C|Z) < Var(Y^T-Y^C) = \sigma^2_T + \sigma^2_C$$

even though under the null hypothesis, conditional upon $Z$, $Y^T$ and $Y^C$ are uncorrelated. Your bolded assumption that $Y^T$ and $Y^C$ are uncorrelated even when not conditioning on $Z$ has the effect of removing any benefit of conditioning upon $Z$, thus giving you the result that $Var(\mathcal{E}_1) = Var(\mathcal{E}_2)$.

If we alter that assumption to $Y^T$ and $Y^C$ are uncorrelated conditional upon $Z$ in your derivation of $Var(\mathcal{E}_2)$, we will have:

$$Var(\mathcal{E}_2)=\frac{1}{N^2}\sum_{i=1}^NVar(Y^T_i-Y^C_i|Z_i)=\frac{1}{N^2}\sum_{i=1}^N(Var(Y^T_i|Z_i) + Var(Y^C_i|Z_i))$$

and substituting $Var(Y^T|Z) \leq \sigma^2_T$ etc. into the last term gives us the inequality:

$$Var(\mathcal{E}_2) \leq \frac{1}{N^2}\sum_{i=1}^N(\sigma^2_T + \sigma^2_C) = {\sigma^2_T+\sigma^2_C \over N} = Var(\mathcal{E}_1)$$

You can see that this is the same result you will get by allowing $Var(Y^T|Z=k) \leq \sigma^2_T$ in the revised next-to-last line of your second derivation given above, which I will repeat for a minor improvement (I hope) in clarity:

$$\frac{1}{N|T|}\sum_{k}Var(Y^T|Z=k)+\frac{1}{N|C|}\sum_{k}Var(Y^C|Z=k)$$

Thus, your two derivations (up to the next-to-last line of the second one) are in agreement, as long as we make sure that both are making the same assumptions about whether conditioning on $Z$ has any effect on the variances.

I think @jbowman gets the maths spot on, and I agree that the final inequality should be ≤. I will give a more descriptive slant to the derivation as the maths is covered well above.

Total variance is the sum of all independent variances plus any covariances present in the dataset. A key assumption you make is zero covariance between Z and $\mathcal{E}$, and the outcome would indicate if that assumption is met or violated. I’m ignoring any variables outside of Z and treatment effect $\mathcal{E}$ here.

In the case of unpaired randomly selected samples you assume that there is no correlation between group assignment and variable Z. This means that with very many samples on average the difference in Z between T and C is 0. However, in individual trails there is a a distinctly finite number of samples and there is a risk of imbalance (inversely related to sample size, you can test this your self by generating independent vectors of random numbers of different lengths and measuring the average correlation achieved over many repetitions). This will inflate the apparent effect size, leading to a positive pressure on the estimated $\mathcal{E}_1$. If there is any breakdown in the assumption that Z and treatment are unrelated then there will be an increase in covariance and then what we would see is that a bias creeps in proportion to the covariance. This means that the difference in Z between T and C will become non-zero, leading to an upwardly biased estimate of $\mathcal{E}_1$ no matter how many trials you undertake.

Now if you pair based on Z you either get 0 difference in all pairs, or you have an intentionally minimised solution to the difference in Z between the pairs. This means that the influence of the confounding element in Z is intentionally reduced as close to zero as possible. This means that when you sum the differences between pairs you get a sum of squares as close to zero as is possible BY DESIGN. If you have designed your algorithm for pairing correctly it should not be possible to get a less biased estimate of $\mathcal{E}$

The overall conclusion is that paired samples derisk having an upwardly biased and over optimistic $\mathcal{E}$. $\mathcal{E}_1$ will at best equal $\mathcal{E}_2$ over many trials if no correlation is present, but will generally exceed $\mathcal{E}_2$ and certainly will if there is a confounding effect, which will artificially inflate the apparent effect size.