2
$\begingroup$

I have some problems understanding the results of this online experiment.

I sent my website users 2 types of messages that can result in a subscription or not.

The users can belong to two different platform categories (iOS / Android).

The results are the following:

Message A Message B
iOS 9.3% (16200 subscriptions / 174000 messages sent) 8,7% (46800/540000)
Android 7.3% (38400 subscriptions / 526000 messages sent) 6.9% (11000 / 160000)
Both 7.8% (54600 subscriptions / 700000 messages sent) 8.3% (57800 / 700000)

Message A seems to be better on iOS and on Android, in terms of subscriptions yielded. But overall message B looks like the better option.

How do I interpret the result?

What kind of test should I perform?

I used the formula as:

  prop.test(x = c(54600, 57800), n = c(700000, 700000))

on the "Both" category but I still am not sure which kind of message could be sent to all users.

Thanks for your help!

$\endgroup$
6
  • 5
    $\begingroup$ en.wikipedia.org/wiki/Simpson's_paradox $\endgroup$ Jan 25, 2022 at 18:23
  • $\begingroup$ @StephanKolassa thank you! this means that the paradox cannot be solved unless I add an extra dimension, i.e. the country of each user, or the gender, or any other confounding variable? at the end, given the data as it is, the problem has not a solution as far as I understand. $\endgroup$ Jan 25, 2022 at 18:30
  • 1
    $\begingroup$ Alternatively, work on balancing your numbers. The problem lies in the fact that for iOS, far more Bs than As were sent, and for Android the reverse. If you can collect more data so the numbers balance out, the paradox should go away. $\endgroup$ Jan 25, 2022 at 18:32
  • $\begingroup$ @StephanKolassa the messages were sent based on the user's country. If I add the country dimension and analyze the data, this could be a way to solve the paradox? Alternatively, there are way to create a simulation in R to balance the samples? $\endgroup$ Jan 25, 2022 at 18:45
  • $\begingroup$ Adding the country to the analysis may be helpful. We can't say without knowing more. There is no way to create data out of thin air to balance your numbers, and I would not "downsample", which is essentially throwing away data. Try analyzing your data with country information. $\endgroup$ Jan 25, 2022 at 20:03

1 Answer 1

0
$\begingroup$

I think this's may be a data record error, you may reverse Android a/b tag: If you have randomized, it is very unlikely to observe such proportions:

  • IOS a/b sample ratio: 174000 / 540000 = 0.32
  • Android a/b sample ratio: 526000 / 160000 = 3.29

If this is by design, regression adjust/stratified average treatment effect may be helpful for estimate Both treatment effect.

y <- rep(0:1, c(4, 4))
trt <- rep(c("A", "B"), 4)
platform <- rep(c("IOS", "IOS", "Android", "Android"), 2)
all_cnt <- c(174000, 540000, 526000, 160000)
subscriptions <- c(16200, 46800, 38400, 11000)
non_subscriptions <- all_cnt - subscriptions

weight <- c(non_subscriptions, subscriptions)

df <- tibble(
  y,
  trt = factor(trt, levels = c("B", "A")),
  platform,
  weight
)

df_long <- df %>% 
  rowwise() %>% 
  mutate(ids = list(seq(1, weight))) %>% 
  unnest(ids)

summary(lm(y ~ trt + platform, df_long)) 

Call:
lm(formula = y ~ trt + platform, data = df_long)

Residuals:
    Min      1Q  Median      3Q     Max 
-0.0923 -0.0869 -0.0733 -0.0733  0.9321 

Coefficients:
            Estimate Std. Error t value            Pr(>|t|)    
(Intercept) 0.067884   0.000527   128.7 <0.0000000000000002 ***
trtA        0.005384   0.000539    10.0 <0.0000000000000002 ***
platformIOS 0.019040   0.000539    35.3 <0.0000000000000002 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.272 on 1399997 degrees of freedom
Multiple R-squared:  0.000962,  Adjusted R-squared:  0.000961 
F-statistic:  674 on 2 and 1399997 DF,  p-value: <0.0000000000000002

for Both: trtA = 0.54%. Message A seems to be better on iOS and on Android, and on Both.

$\endgroup$
1
  • $\begingroup$ thanks! but still, I don't understand how this is significant for "both"... can you help me understand the result of the lm() output? $\endgroup$ Feb 7, 2022 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.