1
$\begingroup$

I checked a few posts on cv, but couldn't get the exact answer. Which test can I use and test if a single proportion (say proportion of choosing a particular item ) from an AB test is significantly different from zero. Here the hypothesis that I am testing is if significantly more people are choosing from the item, when recommended (the other variant is that they are not offered any items). So, wanted to test significance of that proportion vs. 0.

Can I use a one sample Z test ? with population mean as 0 and the sample standard deviation = standard deviation of the proportion that I wanted to test (against 0)

EDIT: actually the test is an email marketing campaign, one campaign variant which has just the viewed item (left in cart) and other that has 3 recommended items along with the viewed item. I wanted to test if significant proportion of people are clicking or buying the recommended items.

Now given this, how can I calculate the statistically significant number of people clicked and viewed (or bought) the recommended items. Also, should if I am testing the significance of the recommended items bookings or views (if the proportion of people viewed are significantly different from zero), should I calculate my sample size based on this (recommended item's conversion) effect size.

$\endgroup$
2
  • 3
    $\begingroup$ If anyone chooses the item, the probability of choosing that item is greater than zero, and you reject your null hypothesis with certainty. This fact makes me wonder if you meant to ask a different question. Some of what you’re writing makes me think you want to test if two proportions are different. Is that what you want to do? $\endgroup$
    – Dave
    May 6, 2020 at 23:46
  • 2
    $\begingroup$ I have seen your edit and think you still need to refine your problem. When no recommended products are shown, there’s no chance for a person to select a recommended product. When the recommended products are shown, there’s a chance that someone will select them. If even one is selected, then you have proof (not evidence, proof) that there’s better than zero chance of a recommended item being selected. $\endgroup$
    – Dave
    May 7, 2020 at 1:53

1 Answer 1

1
$\begingroup$

This sounds more like an estimation problem rather than a hypothesis testing problem. As Dave notes, a single selection of the item in the variant group will contradict the hypothesis that the probability of selection is 0 (which is equivalent to saying it is impossible that anyone would select the item when it is recommended, which while physically possible is statistically unlikely).

Let me demonstrate. The binom library implements several different confidence intervals. Assuming 1 person in 10,000 buys or selects your product, here would be the confidence intervals:

         method x     n        mean         lower        upper
1  agresti-coull 1 10000 0.000100000 -4.282233e-05 0.0006267439
2     asymptotic 1 10000 0.000100000 -9.598660e-05 0.0002959866
3          bayes 1 10000 0.000149985  1.553338e-07 0.0003907725
4        cloglog 1 10000 0.000100000  1.125906e-05 0.0005843805
5          exact 1 10000 0.000100000  2.531778e-06 0.0005570370
6          logit 1 10000 0.000100000  1.408618e-05 0.0007095439
7         probit 1 10000 0.000100000  1.253757e-05 0.0006322640
8        profile 1 10000 0.000100000  7.735773e-06 0.0004402345
9            lrt 1 10000 0.000100000  1.000000e-04 0.0004388696
10     prop.test 1 10000 0.000100000  5.220037e-06 0.0006492406
11        wilson 1 10000 0.000100000  1.765267e-05 0.0005662689

Note all 11 have a lower bound greater than 0, so you would reject in every case. All it takes is 1 person, which could even be an accident for all you know, to reject your null. If a test is that fragile, then it isn't worth performing.

I'm not sure what you're interested in, but you could instead just observe how many conversions are made when recommended. In order to get a confidence interval of radius $r$, you need approximately

$$ n = \left(\dfrac{1}{r}\right)^2 $$

observations. Frankly, probably fewer than this because conversion is likely very small and so the estimated standard deviation is smaller than 0.5 (an approximation to the true standard deviation which I have made here).

All in all, don't test. Instead, estimate.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you Demetri. I have updated my question with details. Sorry for not specifying that earlier $\endgroup$
    – tjt
    May 7, 2020 at 1:30
  • 2
    $\begingroup$ @tjt This sounds like a classic AB test, so you can search the site for similar questions. I know I've answered about a dozen or so. $\endgroup$ May 7, 2020 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.