0
$\begingroup$

We would like to conduct an AB Test between 2 groups of people and the apps that people have to see which ones over or underindex. Which is more appropriate odds ratio or 2 proportional z test and why?

However my concern with the odds ratio is that it would be alot harder to explain odds to a stakeholder than it would be for regular probability.

Raw Data

DeviceId App Audience Group
1 Tinder A
1 Bumble A
1 CNN A
2 Tinder A
2 Bumble A
2 Solitaire A
3 Tinder B
3 Bumble B
3 Disney+ B
4 Disney+ B
4 Tinder B
5 WSJ A
5 Disney+ A
6 WSJ B
6 Solitaire B
6 CNN B

Aggregated you get the below: Group A Aggregated Statistics

Apps Total Devices Percentage 1-Percentage Odds
Tinder 2 67% 33% 2.02
Bumble 2 67% 33% 2.02
Solitaire 1 33% 67% 0.49
CNN 1 33% 67% 0.49
Disney+ 1 33% 67% 0.49
WSJ 1 33% 67% 0.49

Group B Aggregated Statistics

Counts for Apps Total Devices Percentage 1-Percentage Odds Ratio
Tinder 2 67% 33% 2.02
Bumble 1 33% 67% 0.49
Solitaire 1 33% 67% 0.49
CNN 1 33% 67% 0.49
Disney+ 2 67% 33% 2.02
WSJ 1 33% 67% 0.49

Odds Ratio vs Probability Comparison

App Group A Probability Group A 1-Probability Group B Probability Group B 1 - Probability P1/P2 Odds Ratio
Tinder 67% 33% 67% 33% 1 100%
Bumble 67% 33% 33% 67% 2 400%
Solitaire 33% 67% 33% 67% 1 100%
CNN 33% 67% 33% 67% 1 100%
Disney+ 33% 67% 67% 33% 0.5 25%
$\endgroup$

1 Answer 1

0
$\begingroup$

You can use the usual test of the difference between two proportions. You'll need to do it separately for each of the 6 apps. Be mindful of the fact that you're conducting multiple tests; to control the overall Type I error probability, you might consider using a Bonferroni correction.

If you are creating confidence intervals for the differences in probabilities, I suggest using the Agresti-Caffo method.

$\endgroup$
1
  • $\begingroup$ Agree (heart) on the bonferoni correction and the multiple comparison bias. Good call. That has been a back burner thought for a while. $\endgroup$
    – Hider1466
    Commented Jul 9, 2023 at 16:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.