# experiment with 3 arms: analyzing results when one of 3 pairwise tests shows significance

I have an experiment with 3 groups:

• group 1 got no email
• group 2 got an email with 10 dollars incentive
• group 3 got an email with 20 dollars incentive

to convert to a specific product.

Conversion rates are as follows:

• group 1 - 1.2%
• group 2 - 1.3%
• group 3 - 1.6%

I am looking at pairwise significance tests, and I get the following results:

• group 1 vs group 2: p-value 0.6
• group 2 vs group 3: p-value 0.2
• group 1 vs group 3: p-value 0.009 - significant.

I am struggling to interpret these results. It looks like sending an email with 20 dollars is more effective than sending no email, but it is not more effective than sending the 10 dollars email, which in turn is not more effective than sending no email. Contradiction?

I understand p-values are probabilities, so for the first comparison there is a 60% chance to observe those values at random, so I think it's something about probability of intersection of events - but I'm missing something to get the complete picture for this.

Thank you

The thing that you need to understand/remember is that these tests are used to rule out certain cases, but the failure to rule something out does not make it true.

Consider the following claims (hypotheses) about a coin:

1. The coin is 2-headed therefore $p(Heads) = 1.0$.

2. The coin is fair therefore $p(Heads) = 0.5$.

3. The coin is 2-tailed therefore $p(Heads) = 0.0$

4. The coin is biased and $p(Heads) = 0.9$

If I flip the coin and observe Heads, then I cannot rule out option 1, but that does not mean that option 1 is the truth, in fact, with a single flip showing heads I can only rule out option 3. Options 1, 2, and 4 are still possible even though at most 1 of the 3 can be the "Truth". If I flip the coin a total of 10 times and see Heads each time then I can rule out option 2 at standard significance levels (but there it is still possible that 2 is correct, just unlikely). But 10 heads still leaves options 1 and 4 as possibilities.

In your case you have ruled out the possibility that groups 1 and 3 are the same (at a given significance level), but you have not ruled out that 1 and 2 are the same (but that does not prove that they are the same) and you have not ruled out that 2 and 3 are the same (but that does not prove that they are), you have also not ruled out the possibility that 2 is different from both 1 and 3. So from your evidence you are reasonably certain that 1 and 3 are different, but 2 could be the same as either 1 or 3 (but not both at the same time since they are different) or could be different from both, you just don't have enough evidence to rule any of those out. No contradiction unless you make the mistake of treating possibility as certainty.

So, there are a couple things going on here.

First, the phrase 'there is a 60% chance to observe those values at random' is a bit loose to understand whether you are in fact interpreting the results correctly.

When you complete a t-test you assume the null hypothesis to be true as well as some other underlying assumptions to be roughly true. Under these assumptions, the p-value tells you how often you would observe a value as extreme or more extreme than the observed value given repeated sampling. When this value is very low, this suggests that the observed sample value has a small probability of occuring by chance.

So what you can conclude from your tests is that the difference between Group 1 and Group 3 does not seem to have occurred as a result of your particular sample and may represent a true difference between the groups.

However, you cannot conclude there is no difference between Group 1 and Group 2. Rather, you have a lack of evidence for a difference between them. You can't prove the null by failing to reject it. This blog post goes into further detail that might be useful for you on this topic.