I have an experiment with 3 groups:
- group 1 got no email
- group 2 got an email with 10 dollars incentive
- group 3 got an email with 20 dollars incentive
to convert to a specific product.
Conversion rates are as follows:
- group 1 - 1.2%
- group 2 - 1.3%
- group 3 - 1.6%
I am looking at pairwise significance tests, and I get the following results:
- group 1 vs group 2: p-value 0.6
- group 2 vs group 3: p-value 0.2
- group 1 vs group 3: p-value 0.009 - significant.
I am struggling to interpret these results. It looks like sending an email with 20 dollars is more effective than sending no email, but it is not more effective than sending the 10 dollars email, which in turn is not more effective than sending no email. Contradiction?
I understand p-values are probabilities, so for the first comparison there is a 60% chance to observe those values at random, so I think it's something about probability of intersection of events - but I'm missing something to get the complete picture for this.