# Can we do a reasonable statistics interpretation on A/B test if group size is not known

My coworker did A/B test finding a 6.5% diff in two groups conversion rates. First group has 1176 converted events and second group 1309 converted events. The total number of events before conversion was 4094 -- he had randomization to the two arms so my colleague assumed 2047 events in both cases before conversion. 6.5% = 1309 - 1176 / 2047 My argument was that this is unreasonable -- it is more likely that the actual conversion is partially explained by the diff in the data before the conversion -- 2047 is likely to be tilted up in the arm with better observed conversion

However, as there are two unknowns, is there any good way to think around this? Is 6.5% right? 0, or something in between?

• Can you explain this a little more? Jan 18 at 3:37
• Happy to! So typically in an A/B test we know how many people have been pushed to the two branches (A and B). However, in our case, we did not capture that data -- we only recorded events: Regarding events: 1176 events in group A and 1309 events in group B. The difference could be explained by two things: 1) There is actual difference in the groups (our hypothesis) 2) Difference could also be due to poor coinflipping: instead of 50-50 split (2047 on both groups) we might have had 2000 and 2094, explaining majority of the variation. Can we deduce how the two factors co-contributed? Jan 26 at 10:13
• I do not understand the terminology: conversion rates, converted events, arms, branches, events, coinflipping, factors. Pls clarify. Jan 31 at 21:44
• Arm = Branch = Group = A/B: In A/B there are two different groups of individuals Events are the cases we are investigating in the A/B test -- e.g., one death of a patient. See below link for information on ab-testing for further terminology: optimizely.com/optimization-glossary/ab-testing Coinflipping is 50-50 randomization, where 50% of the people are put to one branch / group, while 50% of the people are put to another branch / group. The two factors I am referring to are: 1) randomization to the two groups, and 2) true difference in the groups Feb 4 at 18:38
• Also for those interested, discussed with colleague and in normal statistical sense, there is no way to "split" the variation into the two factors without having any additional information -- the end result could very well have been caused by a poor chance (as we do not know how many people were in the group) or by a real factor. However, using bayesian probabilities, we can apparently use the 50-50% information as additional data, and apparently bayesian gives a distribution that just has thicker tails, but average of 6.5 (same as assuming exact 50-50% split) Feb 4 at 18:42