6
$\begingroup$

I've recently come across a method known as "Controlled-experiment Using Pre-Existing Data" (CUPED), which is a regression adjustments technique to create unbiased estimators in experiments. Likewise, I have prior familiarity with difference-in-differences (DiD).

As I understand it, DiD makes the assumption that treatment groups and control groups are changing at the same rate; the means of treatment and control might differ before the experiment begins. However, we're interested in the change in the treatment group (while accounting for the existing trend) and the change in the control group (only the existing trend).

CUPED seems to accomplish much the same thing. But I've heard that instead of modeling the difference in trends, it tries to account for trends such that in the absence of treatment exposure, the treatment group is corrected to match the control group. And this adjustment carries through the experiment such that treatment and control groups could be comparable directly. Is any of this true?

Further, I have a few questions:

  1. Does CUPED assume non-zero trends exist in both treatment and control groups?
  2. If [1], does CUPED assume the same trend?
  3. Does CUPED's adjustment require data from pre- and post-experiment start (like DiD)? Or does CUPED instead use purely pre-experiment panel data to make the adjustment?
  4. Do both treatment and control groups receive an adjustment?
$\endgroup$

2 Answers 2

3
$\begingroup$

i'm try to answer this question from a practitioner's perspective: in A/B testing, or online controlled experiment, when people mention CUPED, or DID, what are they actually doing and the assumptions reflected in these practices.

for simplicity, we start from a simulated data:

  • y: response variable, or post-experiment data
  • t: assign variable, indicating whether it is a treat group or a control group
  • x: pre-experiment variable, usually is the value of y in pre-experiment period
library(tidyverse)
options(pillar.sigfig = 6)
options(digits = 6)

set.seed(0)
N = 10000 # sample size
p = 0.5 # treat group ratio
tau = 0.2 # true treat effect
theta = 0.6 # correlation between x and y
t = ifelse(runif(N) > p, 1, 0) # assign variable
x = rnorm(N) # pre-experiment variable
y = theta * x + t * tau + rnorm(N) # response variable
df = data.frame(y, t, x)

Diff-in-Diff

in practical A/B testing, some people do use DID because it seems intuitive, although many other people criticize this method as being inappropriate in the context of randomized experiments.

here is summary stat, the raw delta y is 0.197327 - 0.00625198 = 0.191075, pre-exist delta x is -0.00297056 - 0.0151793 = -0.0181499, close but not 0.

df |>
  group_by(t) |>
  summarise(
    n = n(),
    my = mean(y),
    mx = mean(x)
  )
      t     n         my          mx
  <dbl> <int>      <dbl>       <dbl>
1     0  5040 0.00625198  0.0151793 
2     1  4960 0.197327   -0.00297056

DID correct the pre-exist delta x by this: (0.197327 - 0.00625198) - (-0.00297056 - 0.0151793) = 0.209225.

here is so called DID model, also as change score model. the estimate of t is the same as the value we calculated manually before

did_mod = summary(lm(y ~ t, df, offset = x)) # or summary(lm(y - x ~ t, df))
did_mod
did_mod$coef['t', 'Estimate']
    did_mod$coef
               Estimate Std. Error   t value    Pr(>|t|)
(Intercept) -0.00892729  0.0151799 -0.588099 5.56479e-01
t            0.20922526  0.0215540  9.707023 3.52553e-22

in DID model, if without treat effect, y = x + intercept, here intercept is time effect, which is constant for all.

CUPED

the main idea of CUPED is:

  • compute $\theta$

$$\theta = \frac{\operatorname{cov}(Y, X)}{\operatorname{var}(X)} = \operatorname{corr}(X, Y) \cdot \operatorname{var}(Y) = \rho \cdot \operatorname{var}(Y)$$

$\theta$ turns out to be the ordinary least square (OLS) solution of regressing Y on X

theta = cov(df$y, df$x) / var(df$x)
theta
summary(lm(y ~ x, df))$coef['x', 'Estimate']
[1] 0.594981
[1] 0.594981
  • compute adjusted $Y_i^{cv} = Y_i - (X_i - \mu_X) \cdot \theta$ for each user
mx = mean(df$x)
df$y_cv = df$y - theta * (df$x - mx)
  • evaluate the A/B testing using $Y_i^{cv}$ instead of $Y_i$
cuped_mod = summary(lm(y_cv ~ t, df))

the result cuped-adjusted estimate treat effect is

$$\tau = (\overline Y_1 - \theta \cdot (\overline X_1 - \mu_X)) - (\overline Y_0 - \theta \cdot (\overline X_0 - \mu_X)) \\= (\overline Y_1 - \overline Y_0) - \theta \cdot (\overline X_1 - \overline X_0) $$

in our case, $\tau = 0.197327 - 0.00625198 - (-0.00297056 - 0.0151793) * \theta = 0.201874 $, the same as below

cuped_mod = summary(lm(y_cv ~ t, df))
cuped_mod$coef
cuped_mod$coef['t', 'Estimate']
               Estimate Std. Error    t value    Pr(>|t|)
(Intercept) 0.000895777  0.0140728  0.0636532 9.49248e-01
t           0.201874228  0.0199820 10.1028043 6.98895e-24
[1] 0.201874

DID vs CUPED

  • the only difference is $\theta$, in DID, $\theta$ fixed to 1, in CUPED, $\theta = \frac{\operatorname{cov}(Y, X)}{\operatorname{var}(X)}$, there are also other calculation methods.

  • because of the difference in $\theta$, the standard errors of the CUPED and DID estimates are different. Generally, the cuped estimate is more accurate.

  • because of the difference in $\theta$, there are also slight differences in the treatment effects estimated by CUPED and DID. refer to (Best practice when analysing pre-post treatment-control designs)[https://stats.stackexchange.com/questions/3466/best-practice-when-analysing-pre-post-treatment-control-designs]

Back to your question

Does CUPED assume non-zero trends exist in both treatment and control groups? If [1], does CUPED assume the same trend?

yes. $\theta$ is common for both treatment and control groups. But I think we can also estimate $\theta$ in the experimental group and the control group separately.

Does CUPED's adjustment require data from pre- and post-experiment start (like DiD)? Or does CUPED instead use purely pre-experiment panel data to make the adjustment? Do both treatment and control groups receive an adjustment?

CUPED's adjustment require data from pre- and post-experiment start and both treatment and control groups receive an adjustment

$\endgroup$
5
$\begingroup$

The comparison is apples to orangutans, but I will take a stab at it.

CUPED is a variance reduction technique used in experimental analysis of in-test outcomes. It allows you to detect smaller effects by using historical (pre-test) data to adjust the in-test outcome for both groups. This does not give you unbiased estimators of the effect by itself; it seeks to improve sensitivity. Unbiasedness comes from experimental randomization.

DID is useful when there are preexisting time-invariant differences between T and C because treatment was not randomized. It compares pre-post changes for both groups, eliminating the bias from pre-existing differences (when trends are parallel).

$\endgroup$
6
  • $\begingroup$ Perhaps dumb question, if T&C have negligibly different means, pre-exp, do the trends matter? And what is the basis of the correction? $\endgroup$
    – jbuddy_13
    Apr 25 at 5:53
  • 2
    $\begingroup$ CUPED compares in-test adjusted outcomes, so trends are kind of irrelevant since it’s all post data. But the gaps should worry you since any pre-treat differences will likely persist during the experiment and confound the analysis. I am not sure I understand the second question. $\endgroup$
    – dimitriy
    Apr 25 at 6:22
  • $\begingroup$ Perhaps it is more useful to compare DID with ANCOVAs. While CUPED and ANCOVAs are asymptotically equivalent with 50-50 splits, there is more of a literature. For example, see the paper cited here: stats.stackexchange.com/a/325986/7071 $\endgroup$
    – dimitriy
    Apr 25 at 14:17
  • $\begingroup$ I’ll rephrase— what does CUPED do such that pre-exp data is consumed and in-experiment data’s variance is reduced? $\endgroup$
    – jbuddy_13
    Apr 25 at 15:20
  • $\begingroup$ @jbuddy_13 See the ANCOVA formula in the post I linked or arxiv.org/html/2312.02935v1. $\endgroup$
    – dimitriy
    Apr 25 at 15:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.