1
$\begingroup$

I came across this variance reduction technique https://www.exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf

the main idea is using correlation with a pretreatment covariate

$$ \hat{Y}cv = \bar{Y} - \theta (\bar{X} - E(X)) $$

my confusion is on how to estimate below

$$ \theta = \frac{\text{cov}(X, Y)}{\text{var}(X)} $$

$$ \rho = Corr(Y,X) $$

if we use Y (post treatment) ,X (pre treatment) from experiment results, they are a random variable so $\theta$ would also be a random variable. moreover, the treatment and control populations could have different correlations.

what makes sense to me is to get the correlation from pre experiment data where we use consecutive time periods before the experiment and assume it will hold in the experiment period too.

$\endgroup$
3
  • $\begingroup$ why 𝜃 is a random variable is a problem? we can use the estimated value from the pooled population of control and treatment. $\endgroup$
    – wei
    Commented Mar 14, 2022 at 7:55
  • $\begingroup$ Take an example of ecommerce where exposure to some change(treatment) causes some users to increase spend compared to earlier and others to decrease to spend compared to earlier. the pooled estimate is influenced by the treatment. pooling reminds me of the 2 sample t test we check if the variances are similar before taking a pooled variance. in that case we estimate them separately. $\endgroup$
    – dasman
    Commented Mar 14, 2022 at 9:51
  • $\begingroup$ in this case, it seems only influences the variance of 𝜃 , not the point estimate $\endgroup$
    – wei
    Commented Mar 14, 2022 at 10:20

1 Answer 1

0
$\begingroup$

From the point of the original paper, the suggested way is simple estimate it is from the pooled population of control and treatment:

There is a slight subtlety that’s worth pointing out. The pair (Y, X) may have different distributions in treatment and control when there is an experiment effect. For ∆cv to be unbiased, the same θ has to be used for both control and treatment. The simplest way to estimate it is from the pooled population of control and treatment. The impact on variance reduction will likely be negligible

As far as I know, most data scientists, or AB experimental platforms of technology companies, also use this approach.

According to your comment, the underlying question seems to be:

  • why simple estimating the pooled population of control and treatment correlation is reasonable?
  • do we need to do a test of homogeneity of correlation first?

My answer to these questions is yes. This leads us to a more classical variance reduction method: ANCOVA or regression adjustment, which requires the assumption of homogeneity of regression slopes and has a more reliable way of pooling the slopes of control/treatment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.