# On improving the sensitivity of controlled experiments by utilizing pre-experiment data (CUPED)

I’m currently reading “Improving the sensitivity of online controlled experiments by utilizing pre-experiment data” by Deng et al. and struggling to derive a few equations from the paper. I would appreciate some help from those who are familiar with the paper.

The first problem is with the first equation on page 4:

$$\text{var}(\hat{Y}_\text{strat}) = \sum_{k = 1}^K \frac{w_k}{n} \sigma^2_k.$$

It does not seem to match the one given on Wikipedia:

$$s_{\bar {x}}^{2}=\sum _{h=1}^{L}\left({\frac {N_{h}}{N}}\right)^{2}{\frac {s_{h}^{2}}{n_{h}}}.$$

Presumably, $$w_k = N_h/N$$ and $$\sigma^2_k = s_h^2/n_h$$, but then $$w_k$$ is not squared, and the equation has an extra $$n$$ in the denominator.

The second problem is with equation 5 on the same page. How does one substitute

$$\theta = \frac{\text{cov}(X, Y)}{\text{var}(X)}$$

into

$$\text{var}(\hat{Y}_\text{cv}) = \frac{1}{n}(\text{var}(Y) + \theta^2 \text{var}(X) - 2 \theta \text{cov}(X, Y))$$

to get

$$\text{var}(\hat{Y}_\text{cv}) = \text{var}(\bar{Y})(1 - \rho^2)?$$

A sheet of paper and a pencil do not seem to help. Thank you!

• First question: you were probably missing the definition of $n_k$ before equation (2). – Yair Daon Nov 27 '19 at 20:55

First question:

\begin{align} \begin{split} Var(\hat{Y}_{strat}) &= Var(\sum_k w_k \bar{Y}_k) \\ &=\sum_k w_k^2 Var(\frac{1}{n_k}\sum Y^{(k)}_i) \\ &=\sum_k w_k^2 \frac{1}{n_k^2} n_k Var(Y^{(k)}_1)\\ &=\sum_k w_k^2 \frac{1}{n_k}\sigma_k^2\\ &=\sum w_k^2 \frac{1}{n w_k} \sigma_k^2\\ &=\sum w_k \frac{\sigma_k^2}{n}\\ \end{split} \end{align}

You were probably missing the definition of $$n_k$$ above equation (2).

Second question:

\begin{align} \begin{split} Var(Y) + \theta^2 Var(X) -2\theta Cov(X,Y) &= Var(Y) + \frac{Cov^2(X,Y)}{Var^2(X)} Var(X) - 2\frac{Cov(X,Y)}{Var(X)} Cov(X,Y)\\ &=Var(Y) - \frac{Cov^2(X,Y)}{Var(X)}\\ &=Var(Y)(1-\frac{Cov^2(X,Y)}{Var(X)Var(Y)})\\ &=Var(Y)(1-Corr^2(X,Y)) \end{split} \end{align}

Then divide by $$n$$ to get the variance of the sample mean.

• Right! In the first, I was indeed missing $n_k = w_k n$, and in the second, I actually got it, but for some reason, I failed to see it was correct. Thank you! – Ivan Nov 28 '19 at 6:11
• You're welcome. Vote up and give me the bounty please. – Yair Daon Nov 28 '19 at 8:18
• Sure! It’s just not possible within 24 since the bounty start. – Ivan Nov 28 '19 at 11:03