5
$\begingroup$

I need some guidance on how to analyse the results of a subgroup breakdown in an A/B test.

I have the results of an (ongoing) A/B test and need to do an interim analysis on

  1. The overall headline results
  2. The breakdowns by relevant dimensions.

The result is complicated by the fact that the results for the effect-size on some of the broken-down groups are wildly different to the headline (overall) results.

I'm 90% certain that what we are seeing is an artifact of us breaking down the overall allocations into these different groups, i.e. it's Simpson's paradox.

For example we have numbers which look like this:

We have ttwo groups: A, B with the following allocations and conversions (these are not the real numbers, but illustrative):

A B
Mobile allocated 200,000 200,000
Mobile converted 10,000 10,775
Desktop allocated 50,000 50,000
Desktop allocated 2500 2350

I.e. we're seeing an overall lift of 7.75% but a decrease of ~6% on Desktop.

Are there any results on confidence intervals or ways of relating the empirical means and standard deviations of the subgroups to the overall mean, so I could rule out some kind of further analysis?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

This is an interesting sub-population effect, but it's not Simpson's Paradox. Simpson's Paradox occurs when there is a trend in each group of data individually, but the trend reverses in the full dataset. Here, we don't have Simpson's Paradox - the trend has opposite directionalities in either group, and the overall trend is simply the trend in the numerically larger group. There is no paradox here, we simply observe that the overall trend is the same trend we observed in the majority of the data, which is rather intuitive.

For it to be Simpson's paradox, you'd have to see an increase in each of Mobile and Desktop individually, but see a decrease overall when combining them (or the reverse).

I'll also add that Simpson's Paradox, in cases where it exists, is not typically an "artifact" in the sense of some kind of spurious signal. Rather, it represents that there really is some confounding variable that you should take into account.

Improve this answer
$\endgroup$
7
  • 2
    $\begingroup$ What you are seeing is that the sample size was estimated to provide adequate precision or power in the overall result, implying that it is inadequate for subgroup results which are just noisy. $\endgroup$
    – Frank Harrell
    Commented Aug 23, 2023 at 14:18
  • 2
    $\begingroup$ @FrankHarrell I would agree that subpopulation analysis can often stray into noisy underpowered territory, but is that the case here? The sample size in both subgroups is sufficiently large to identify significant changes of the magnitude seen here. The chi-squared p-value for Desktop alone is 0.03, and for Mobile is <0.00001 - this strikes me as good evidence that B really is better on Mobile, but worse on Desktop. In what sense do you consider this noise? $\endgroup$
    – Nuclear Hoagie
    Commented Aug 23, 2023 at 14:34
  • 1
    $\begingroup$ That's easily determined by generating a random 10-category with equal proportions variable and stratifying the analysis by this variable. You know the truth: the random number cannot be related to Y. Any variation you get must be random by definition. Compare that amount of variation to the amount you witnessed with real data. $\endgroup$
    – Frank Harrell
    Commented Aug 23, 2023 at 14:54
  • 1
    $\begingroup$ @TomKealy I think it comes down the sample size of your subgroups. You have sufficient data here to indicate that B doing worse on Desktop is very likely not the result of random chance. How much each subgroup differs from the overall population is rather irrelevant, the only thing that matters is how well you're quantifying each group individually. If you collect tons of data of some effect that behaves differently in Men and Women, for example, you can accurately quantify the different effect in both groups, neither is "noise" because they differ from the population-wide trend. $\endgroup$
    – Nuclear Hoagie
    Commented Aug 23, 2023 at 15:09
  • 1
    $\begingroup$ Basically, if you have enough data to adequately quantify the effect in Desktop cases (which you do), no amount of data on Mobile cases can invalidate that. It can swamp the effect in an overall population analysis, but a well-powered significant analysis in the subgroup won't go away just because you're collecting more data on cases that are expected to be (and indeed are observed to be) different. $\endgroup$
    – Nuclear Hoagie
    Commented Aug 23, 2023 at 15:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.