# How do I know if I'm seeing Simpson's paradox?

I need some guidance on how to analyse the results of a subgroup breakdown in an A/B test.

I have the results of an (ongoing) A/B test and need to do an interim analysis on

2. The breakdowns by relevant dimensions.

The result is complicated by the fact that the results for the effect-size on some of the broken-down groups are wildly different to the headline (overall) results.

I'm 90% certain that what we are seeing is an artifact of us breaking down the overall allocations into these different groups, i.e. it's Simpson's paradox.

For example we have numbers which look like this:

We have ttwo groups: A, B with the following allocations and conversions (these are not the real numbers, but illustrative):

A B
Mobile allocated 200,000 200,000
Mobile converted 10,000 10,775
Desktop allocated 50,000 50,000
Desktop allocated 2500 2350

I.e. we're seeing an overall lift of 7.75% but a decrease of ~6% on Desktop.

Are there any results on confidence intervals or ways of relating the empirical means and standard deviations of the subgroups to the overall mean, so I could rule out some kind of further analysis?