Need help explaining apparent conflicting results - We ran 2 different, but similar intervention type trials, each with their own control groups. Results were as follows (lower is better):

Trial 1:  Treated n1=936,539   Treated conversion x1=52,223---->5.58%
          Control c1=124,599   Control conversion cx1=6,988---->5.61%
                          Lift= -0.6%(insignificant)

Trial 2:  Treated n2=463,308   Treated conversion x2=116,816--->25.21%
          Control c2=52,218   Control conversion cx2=13,617---->26.08%
                          Lift= -3.3%

Now if we were to combine the two trial to determine is there was a combined intervention effect we get:

Combined:  Treated n=1,399,837   Treated conversion x=169,039--->12.08%
          Control c=176,817   Control conversion cx=20,605---->11.65%

So the combined effect is not showing a positive impact of the combined interventions, although the the two trials are either showing a positive effect or neutral.

Any ideas to describe this phenomenon?

  • 3
    $\begingroup$ Maybe this due to something called the Simpson's paradox when pooling groups gives and overall effect whose sign is different from those of the effects in each separate group. See en.wikipedia.org/wiki/Simpson%27s_paradox $\endgroup$ – winperikle May 28 '19 at 15:29

As @winperikle noted, this is likely a case of Simpson's paradox where combining two disparate groups means a different sign of the overall effect than for each group separately.

From your summary it is obvious that the two experiments are not really comparable as the base rates are very different. It is generally a bad idea to just add together data from two different experiments and it is better to model the experiment as a predictor of the outcome. Hierarchical models might be of big advantage in this case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.