My questions is a bit general, I'm mostly looking for guidance on terms to look for or links/articles to methodologies that map to this example...

Consider a simple online experiment that was meant to test one variant against the control.

  • Variant A = Make button blue.
  • Control = Keep button grey
  • H_0: % of users that click the button does not change if you make the button blue

After launching the experiment, you realize a group of the users (say all users that view on Safari) are actually receiving a different experience, Variant B = Button Green

Group Control Variant_A Variant_B
A     50%     50%       
B     50%               50%

My questions are

  1. Can you perform hypothesis testing on the two groups separately? First consider Group A and make the null hypothesis something about "for all non-safari users, a blue button does not change p" and second, consider Group B with an analogous null.

  2. Can you draw conclusions about which variant is better overall in this case?

I think the test statistic is something like if the difference of the relative lifts. The logic being that if the impact of a colored button for Group A is a 400% lift (say moving CTR from 5% to 25%) and the lift for Group B is only 33% lift (3% to 4% but still stat sig) you might conclude that either

(1) Blue is a much better color than Green overall


(2) that your Group A users respond much better to color than Group B users


(3) we don't know which color converts better overall, we only know that for Group A, Blue converts better than Grey and for Group B, Green converts better than Grey.


I initially answered using logistic regression, but there is little value over just doing pairwise comparisons, in my opinion.

Assuming your experiment was designed so that users experience 1 of 3 variants (Red, Green, or control. If they are experiencing a third when the experiment was not designed to, then you have a different problem) then we can set this up as a logistic regression. Here might be some of the data you see.

# A tibble: 3 x 3
  version  clicks total
  <chr>     <int> <int>
1 _Control    389  3418
2 _Green      513  3289
3 _Red        468  3293

Using R's pairwise functions...

pairwise.prop.test(d$clicks, d$total)

    Pairwise comparisons using Pairwise comparison of proportions 

data:  d$clicks out of d$total 

  1       2     
2 1.5e-06 -     
3 0.0012  0.1227

P value adjustment method: holm 

So in this case, both versions do better as compared to the control (first column, both entries are <0.05) but we can not conclude that green did better than red (second column, p>0.05).

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.