Clustering and A/B testing

Question

My question is the following: Let's imagine I've defined clusters in my data (different segments of customers) and I run an A/B test. Can I compare the performances of the different clusters on the A/B test? I did not find a lot of litterature on it (in fact really close to none) so I was wondering if there was a statistical reason not to do it ?

Here is a detailed explanation of the problem:

Let's imagine I run an A/B test. It turns out that neither A nor B is statistically significantly better than the other. Still, it would be great to derive insights from it. Maybe a subset of the population prefers the new version B and another subset prefers the version A. Let's say I've already determined clusters among my customers, I would like to see how those clusters were affected by the A/B test. For instance people under 20 years old (cluster A) convert 10% more on version B, and people older than 50 (cluster B) convert 10% less. Then, our A/B test that previously was saying that the change was not bringing statistically significant change gives us more insights. We can try to understand why the version B fits more for younger people and less for older ones. We gained some insights from our test. And it can also apply to tests that say that one version is better than another.

Of course, if you do it like this you will very likely find clusters that perform better (or worse) than others. So you would have to run another A/B test on a given cluster, in order to verify your hypothesis.

I have not found other people doing that, is there a statistical reason not to do it or is it a legitimate way of gaining insights ?

Thanks a lot !

Answer 1

Absolutely, you can compare the different clusters, although it's important that you carefully consider what you infer from statistical significance. While it is indeed a very good indicator, by its very nature a threshold of $p<0.05$ will mean that $1/20$ tests will result in a false positive leaving many engineers and scientists to exclaim that an effect is present when it is possibly not. Also if the test returns $p=0.055$ would you immediately conclude that there is no relationship there?

This question touches on the problem of multiple comparisons where the more tests you apply the more likely you are to find something statistically significant. There are simple corrections such as bonferroni which essentially reduces the threshold to $p<0.05/n_{tests}$ although this must be used with caution as it is a rather aggressive correction.

So there is no harm in looking at your data from a number of ways to extract insight from it, in fact I would encourage it. The best advice I could give is to look at your data, plot it out, look at the distributions, how many data points do you have, are they normal or non-parametric or skewed. Get a real feeling of whats going on rather than just relying on statistical tests. If you have a hunch and the p-value looks in the right ball park, gather more data and see if this confirms your theory.