1
$\begingroup$

So we introduced a new feature in our app, that would aid conversion (hypothetically). When I tried to measure this incremental change in conversion, I split my base set of customers into control (C) 30% and test (T) 70% sets via random sampling. Though feature was live to all customers for more than 2 months, we started an experiment say from Aug 19th (t1) where showed the feature only to the customers in test and control was not shown this feature.

What I observed is, for more than 1 month time period before t1 conversion of C was say 29.5%. And that of T's was 29% (both are averaged conversion calculated over 1 month before experiment start).

Now after Aug 19th (after t1) C's conversion was say 31% and T's conversion was say 31.2% (measured over 1-2 weeks after exp started).

Now, I want to calculate the improvement in conversion as follows:

Before t1: Diff in conversion (test-control) = 29-29.5 = -0.5%

After t1: Diff in conversion (test-control) = 31.2-31 = 0.2%

So change in conversion was 0.2-(-0.5) = 0.7%

So test added to an improved 0.7% conversion

Is this the right way to calculate when control set was already having a biased better performance than experiment set before start of experiment? (Assume i will run this exp for few more weeks & also do significance test further, but assume it was significant)

Some more info: The hoped improvement was also in the range of 1% itself. Plus, # of times control doing better than test before experiment start(t1) was about 67% now its come down to about 40%. Hence I am inclined to strongly argue test is adding ~0.7% value.

Improve this question
$\endgroup$
2
  • $\begingroup$ You don't need a difference in difference. You randomly assigned users to treatment and control, so their baseline difference is 0 in expectation. All you need to do is compare post assignment conversion rates. $\endgroup$
    – Demetri Pananos
    Commented Sep 2, 2023 at 12:51
  • 1
    $\begingroup$ You just discovered that there is random variation in your two samples. That's fine and that being said, that said, DiD is fine to use. Using a quasi-experimental methodology with RCT/AB data doesn't invalidate the findings. $\endgroup$
    – usεr11852
    Commented Sep 2, 2023 at 17:29

1 Answer 1

Reset to default
1
$\begingroup$

I see a lot of people randomly assign users to groups and then do a difference in difference. You don't need to do this, and there is probably some loss of statistical efficiency through spending extra degrees of freedom estimating the additional parameters for the DiD (though with big enough samples it may not matter much).

Prior to $t_1$, your users are part of the same population, so the distribution of their future outcomes is the same. Hence, the conversion rate for both treatment and control prior to $t_1$ is the same, meaning the null is true. No need to estimate the difference; we know its 0.

Were these users from different countries, or states, or something like that then DiD would make more sense. I think the canonical example for explaining DiD is the change in minimum wage in New Jersey and Pennsylvania. One of these states changed their minimum wage, the other did not. Because differences existed at baseline (the two states are not identical and have state level differences), you need to adjust for that difference first before estimating the treatment effect. Had we randomized people living in both states -- like AB tests often do -- you don't need that first step and can compare outcomes immediately.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ But the difference was not 0. Makes the assumption invalid & 0.5% difference is significant. Conceptually of-course it makes sense and when we are dealing with say changes in the range of 4-5% then this 0.5% can be ignored.. But graph clearly reduces this variation after exp start and keeps it consistent. $\endgroup$
    – Narahari B M
    Commented Sep 3, 2023 at 14:37
  • $\begingroup$ @NarahariBM the difference is 0 in expectation. Very different thing. $\endgroup$
    – Demetri Pananos
    Commented Sep 3, 2023 at 14:37
  • 1
    $\begingroup$ Indeed expectation is 0, but when we are actually trying to measure it, should we still go with expectation ignoring whats observed? Also doesn't this come under experiment calibration? $\endgroup$
    – Narahari B M
    Commented Sep 3, 2023 at 14:44
  • $\begingroup$ Maybe doing a stratified sampling is probably the right way to solve this. But then i cant just take 1 stratified sample. Need to take multiple and measure it. This was atleast on entire population so thought was not an issue. $\endgroup$
    – Narahari B M
    Commented Sep 3, 2023 at 14:46
  • $\begingroup$ @NarahariBM You're making it too complex, it is absolutely fine to just look at post treatment outcomes due to randomization. As per my answer, it isn't that DiD is wrong, it just isn't needed because of randomization. You can easily see this by looking at the estimator here and propogating the expectation operator. That there was random variation in the pre-treat period is to be expected and changes nothing. $\endgroup$
    – Demetri Pananos
    Commented Sep 3, 2023 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.