4
$\begingroup$

I had a formative experience a couple years ago. The context was a live user experiment where the baseline proportion was approximately 98%. While the strategy had limited room to scale, management was interested in pumping this number as close to 100% as possible; a 2% lift was virtually impossible but certainly the North Star. But in reality, we wanted to detect an effect as small as 0.5%.

At any rate, our company's randomization system would randomly assign users, based on their first login time after experiment start to either the treatment group or control group. Following, the actual treatment exposures began.

In a fluke of randomization, the treatment group's pre-experiment proportion was 97.95% and the control group's was 98.05%. So pre-treatment, the difference between treatment and control groups was 0.1% (a fifth of what we intended to detect, given an adequately powered experiment.)

The conclusion from the experiment was that we could not reject the null hypothesis, that an effect likely did not exist in the population. However, this was instantly scorned by management.

My take at the time was, there's a 0% chance that the pre-treatment, treatment and control group means are exactly even, there will always be some difference but it's a question of decimal points.

I argued that if we had repeated the randomization an arbitrarily large number of times, the difference between means (pre-experiment) would be centered on zero with some variance. The apparent difference that we observed was simply random noise. We could hold that pre-experiment, the each group had a mean of 98.0% that was obscured by random noise.

Management didn't buy this. And I've wondered since then...

  1. Was my conclusion sound?
  2. How would I have communicated the correct conclusion better to a business audience?
Improve this question
$\endgroup$
2
  • $\begingroup$ Just so I understand, did management believe that since a pre-treatment difference existed then the results from the experiment were somehow invalid? Not really clear on what their objections were. $\endgroup$
    – Demetri Pananos
    Commented May 11 at 1:50
  • $\begingroup$ @DemetriPananos, yes you understand correctly. We observed that pre-exp, the control group was a little greater than the Treatment. The exp conclusion was that no effect existed. However, management tried to argue that such an effect did exist, but it washed out due to poor randomization. $\endgroup$
    – jbuddy_13
    Commented May 11 at 6:41

1 Answer 1

Reset to default
3
$\begingroup$

  1. Your conclusion is/was correct. Randomization is not about balancing groups. Indeed, there is a good chance that unobserved confounders are unbalanced and yet randomization still works. In fact, the statistical methods we use are designed to account for this.

  2. This is hard because to really get why this isn't a problem you need a little statistics. Depending on what metric you are estimating in your experiment, you could try to assuage their fears by using past data as control variates (this goes by ANCOVA or CUPED depending on the circles in which you travel).

This article about myths of randomization in clinical trials my be interesting to read

Improve this answer
$\endgroup$
2
  • $\begingroup$ regarding 1/, this might be an obvious question. But if some imbalance in x exists, the MDE must be >x correct? $\endgroup$
    – jbuddy_13
    Commented May 11 at 6:48
  • 1
    $\begingroup$ @jbuddy_13 no! Remember, randomization is not about balancing, so any criterion for balance is a red herring. The MDE is not at all related to pre experiment differences if you are randomizing. $\endgroup$
    – Demetri Pananos
    Commented May 11 at 14:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.