# How to determine if the treatment effect is stronger in one group vs another

My questions is a bit general, I'm mostly looking for guidance on terms to look for or links/articles to methodologies that map to this example...

Consider a simple online experiment that was meant to test one variant against the control.

• Variant A = Make button blue.
• Control = Keep button grey
• H_0: % of users that click the button does not change if you make the button blue

After launching the experiment, you realize a group of the users (say all users that view on Safari) are actually receiving a different experience, Variant B = Button Green

Group Control Variant_A Variant_B
A     50%     50%
B     50%               50%


My questions are

1. Can you perform hypothesis testing on the two groups separately? First consider Group A and make the null hypothesis something about "for all non-safari users, a blue button does not change p" and second, consider Group B with an analogous null.

2. Can you draw conclusions about which variant is better overall in this case?

I think the test statistic is something like if the difference of the relative lifts. The logic being that if the impact of a colored button for Group A is a 400% lift (say moving CTR from 5% to 25%) and the lift for Group B is only 33% lift (3% to 4% but still stat sig) you might conclude that either

(1) Blue is a much better color than Green overall

OR

(2) that your Group A users respond much better to color than Group B users

OR

(3) we don't know which color converts better overall, we only know that for Group A, Blue converts better than Grey and for Group B, Green converts better than Grey.

I initially answered using logistic regression, but there is little value over just doing pairwise comparisons, in my opinion.

Assuming your experiment was designed so that users experience 1 of 3 variants (Red, Green, or control. If they are experiencing a third when the experiment was not designed to, then you have a different problem) then we can set this up as a logistic regression. Here might be some of the data you see.

# A tibble: 3 x 3
version  clicks total
<chr>     <int> <int>
1 _Control    389  3418
2 _Green      513  3289
3 _Red        468  3293


Using R's pairwise functions...

pairwise.prop.test(d$$clicks, d$$total)

Pairwise comparisons using Pairwise comparison of proportions

data:  d$$clicks out of d$$total

1       2
2 1.5e-06 -
3 0.0012  0.1227