1
$\begingroup$

My coworker did A/B test finding a 6.5% diff in two groups conversion rates. First group has 1176 converted events and second group 1309 converted events. The total number of events before conversion was 4094 -- he had randomization to the two arms so my colleague assumed 2047 events in both cases before conversion. 6.5% = 1309 - 1176 / 2047 My argument was that this is unreasonable -- it is more likely that the actual conversion is partially explained by the diff in the data before the conversion -- 2047 is likely to be tilted up in the arm with better observed conversion

However, as there are two unknowns, is there any good way to think around this? Is 6.5% right? 0, or something in between?

$\endgroup$
7
  • $\begingroup$ Can you explain this a little more? $\endgroup$
    – Joel W.
    Jan 18 at 3:37
  • $\begingroup$ Happy to! So typically in an A/B test we know how many people have been pushed to the two branches (A and B). However, in our case, we did not capture that data -- we only recorded events: Regarding events: 1176 events in group A and 1309 events in group B. The difference could be explained by two things: 1) There is actual difference in the groups (our hypothesis) 2) Difference could also be due to poor coinflipping: instead of 50-50 split (2047 on both groups) we might have had 2000 and 2094, explaining majority of the variation. Can we deduce how the two factors co-contributed? $\endgroup$ Jan 26 at 10:13
  • $\begingroup$ I do not understand the terminology: conversion rates, converted events, arms, branches, events, coinflipping, factors. Pls clarify. $\endgroup$
    – Joel W.
    Jan 31 at 21:44
  • $\begingroup$ Arm = Branch = Group = A/B: In A/B there are two different groups of individuals Events are the cases we are investigating in the A/B test -- e.g., one death of a patient. See below link for information on ab-testing for further terminology: optimizely.com/optimization-glossary/ab-testing Coinflipping is 50-50 randomization, where 50% of the people are put to one branch / group, while 50% of the people are put to another branch / group. The two factors I am referring to are: 1) randomization to the two groups, and 2) true difference in the groups $\endgroup$ Feb 4 at 18:38
  • $\begingroup$ Also for those interested, discussed with colleague and in normal statistical sense, there is no way to "split" the variation into the two factors without having any additional information -- the end result could very well have been caused by a poor chance (as we do not know how many people were in the group) or by a real factor. However, using bayesian probabilities, we can apparently use the 50-50% information as additional data, and apparently bayesian gives a distribution that just has thicker tails, but average of 6.5 (same as assuming exact 50-50% split) $\endgroup$ Feb 4 at 18:42

1 Answer 1

0
$\begingroup$

Perhaps your situation is captured by a 2x2 contingency table, with these the two variables: event (yes or no) and group (A or B). If so, a chi sq test would be appropriate.

$\endgroup$
4
  • $\begingroup$ This was my approach first. However, the problem is we do not know the group sizes (A / B). We can of course assume a 50-50 split between the groups, but it is also possible that there is a 60-40 split in the data. Discussed with a colleague that the way to get best estimate is to use bayesian simulation, that takes in to assumption of how the people are divided into the two groups (A/B) -- rather than forcing the 50-50 as an assumption to the contingency table $\endgroup$ Feb 25 at 19:25
  • $\begingroup$ I am still not sure I understand the basic issue that you are concerned about. Is your issue captured by the following example. You have a group of people who died due to food poisoning while on vacation overseas. You know the number of deaths of people who visited country A and the number of deaths of people who visited country B. You want to know if the vacation country is related to the death rate but you do not know how many people visited each country. $\endgroup$
    – Joel W.
    Feb 28 at 15:35
  • $\begingroup$ That's exactly the issue -- well phrased :) $\endgroup$ Mar 2 at 7:46
  • $\begingroup$ @AnttiHemilä I think the best you can do is to estimate the number of people who visited each country based on whatever other research data you have or you can find in the published literature. $\endgroup$
    – Joel W.
    Mar 3 at 2:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.