I need some guidance on how to analyse the results of a subgroup breakdown in an A/B test.
I have the results of an (ongoing) A/B test and need to do an interim analysis on
- The overall headline results
- The breakdowns by relevant dimensions.
The result is complicated by the fact that the results for the effect-size on some of the broken-down groups are wildly different to the headline (overall) results.
I'm 90% certain that what we are seeing is an artifact of us breaking down the overall allocations into these different groups, i.e. it's Simpson's paradox.
For example we have numbers which look like this:
We have ttwo groups: A, B with the following allocations and conversions (these are not the real numbers, but illustrative):
I.e. we're seeing an overall lift of 7.75% but a decrease of ~6% on Desktop.
Are there any results on confidence intervals or ways of relating the empirical means and standard deviations of the subgroups to the overall mean, so I could rule out some kind of further analysis?