My questions is a bit general, I'm mostly looking for guidance on terms to look for or links/articles to methodologies that map to this example...
Consider a simple online experiment that was meant to test one variant against the control.
- Variant A = Make button blue.
- Control = Keep button grey
- H_0: % of users that click the button does not change if you make the button blue
After launching the experiment, you realize a group of the users (say all users that view on Safari) are actually receiving a different experience, Variant B = Button Green
Group Control Variant_A Variant_B A 50% 50% B 50% 50%
My questions are
Can you perform hypothesis testing on the two groups separately? First consider Group A and make the null hypothesis something about "for all non-safari users, a blue button does not change p" and second, consider Group B with an analogous null.
Can you draw conclusions about which variant is better overall in this case?
I think the test statistic is something like if the difference of the relative lifts. The logic being that if the impact of a colored button for Group A is a 400% lift (say moving CTR from 5% to 25%) and the lift for Group B is only 33% lift (3% to 4% but still stat sig) you might conclude that either
(1) Blue is a much better color than Green overall
(2) that your Group A users respond much better to color than Group B users
(3) we don't know which color converts better overall, we only know that for Group A, Blue converts better than Grey and for Group B, Green converts better than Grey.