1
$\begingroup$

At my work, we have this algorithm of sorts, called 'Mean 2.0' for analyzing experiment results. The idea is that your data is divided into four buckets by the partitions, pre vs post experiment start and treatment vs control. So pre-treat, post-treat, pre-ctrl, and post-control are the four buckets.

The algorithm pools the data, marginalizing the treatment vs control differences, and infers a linear regression model, mapping the pre-x values to post-x values. The goal of this algorithm is to adjust for whatever trend was already occurring and continued to do so during the experiment; the outputs are buckets of treatment and control data where the effect of the trend has been removed, so it does not interfere with a NHST confidence intervals, p-values, etc.

To me, this sounds fishy at best. Has anyone ever heard of this or used it in practice? Is this a principled approach or a little too handwavy?

$\endgroup$
1
  • $\begingroup$ When I worked for a marketing company, we did something similar, though I can't remember the details exactly. Do you have details on the linear model used? $\endgroup$ Commented Jan 3, 2022 at 22:37

1 Answer 1

1
$\begingroup$

This works but is not rigorous. In some special implementations, there is a loss of precision.

Let's make some fake data.

set.seed(0)
sample_cnt <- 100000
df <- tibble(
  trt = sample(0:1, replace = T, size = sample_cnt), ## random assign
  y_conversion_rate_pre = runif(sample_cnt, 0.0, 0.3) # conversion_rate pre
) %>% 
  rowwise() %>% 
  mutate(
    y_pre = y_conversion_rate_pre,
    ## assume group A has a 3% increase in conversion rate
    y_post = if_else(trt == 1, y_conversion_rate_pre + 0.03, y_conversion_rate_pre)
  ) %>% 
  rowwise() %>% 
  mutate(
    y_pre = rbinom(1, 1, y_pre), 
    y_post = rbinom(1, 1, y_post)
  ) %>% 
  dplyr::select(trt, y_post, y_pre)
df

the four buckets are:

trt     n  y_pre  y_post
0   50049   7463    7491    
1   49951   7443    8772    

Mean 2.0 = 8772 / 49951 - 7491 / 50049 - (7443 / 49951 - 7463 / 50049) = 0.0260466

Let's do a regression with y diff (y_post - y_pre):

mod <- lm((y_post - y_pre) ~ trt, df)
mod$coefficients['trt']
0.0260466

The estimated value is exactly the same.

In practical application of some company. ab test is split by some hash mod, for example, hash(user_id) % = 1 as treat, other as control, then the four buckets is:

  • pre-ctl: hash(user_id) % = 0 and is_post = 0
  • pre-trt: hash(user_id) % = 1 and is_post = 0
  • post-ctl: hash(user_id) % = 0 and is_post = 1
  • post-ctl: hash(user_id) % = 1 and is_post = 1

This will reduce experimental precision: it includes many samples that are not relevant to the experiment(user only appeared in pre-duration).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.