How to test if a single proportion is significantly different from zero

Question

I checked a few posts on cv, but couldn't get the exact answer. Which test can I use and test if a single proportion (say proportion of choosing a particular item ) from an AB test is significantly different from zero. Here the hypothesis that I am testing is if significantly more people are choosing from the item, when recommended (the other variant is that they are not offered any items). So, wanted to test significance of that proportion vs. 0.

Can I use a one sample Z test ? with population mean as 0 and the sample standard deviation = standard deviation of the proportion that I wanted to test (against 0)

EDIT: actually the test is an email marketing campaign, one campaign variant which has just the viewed item (left in cart) and other that has 3 recommended items along with the viewed item. I wanted to test if significant proportion of people are clicking or buying the recommended items.

Now given this, how can I calculate the statistically significant number of people clicked and viewed (or bought) the recommended items. Also, should if I am testing the significance of the recommended items bookings or views (if the proportion of people viewed are significantly different from zero), should I calculate my sample size based on this (recommended item's conversion) effect size.

Answer 1

This sounds more like an estimation problem rather than a hypothesis testing problem. As Dave notes, a single selection of the item in the variant group will contradict the hypothesis that the probability of selection is 0 (which is equivalent to saying it is impossible that anyone would select the item when it is recommended, which while physically possible is statistically unlikely).

Let me demonstrate. The binom library implements several different confidence intervals. Assuming 1 person in 10,000 buys or selects your product, here would be the confidence intervals:

         method x     n        mean         lower        upper
1  agresti-coull 1 10000 0.000100000 -4.282233e-05 0.0006267439
2     asymptotic 1 10000 0.000100000 -9.598660e-05 0.0002959866
3          bayes 1 10000 0.000149985  1.553338e-07 0.0003907725
4        cloglog 1 10000 0.000100000  1.125906e-05 0.0005843805
5          exact 1 10000 0.000100000  2.531778e-06 0.0005570370
6          logit 1 10000 0.000100000  1.408618e-05 0.0007095439
7         probit 1 10000 0.000100000  1.253757e-05 0.0006322640
8        profile 1 10000 0.000100000  7.735773e-06 0.0004402345
9            lrt 1 10000 0.000100000  1.000000e-04 0.0004388696
10     prop.test 1 10000 0.000100000  5.220037e-06 0.0006492406
11        wilson 1 10000 0.000100000  1.765267e-05 0.0005662689

Note all 11 have a lower bound greater than 0, so you would reject in every case. All it takes is 1 person, which could even be an accident for all you know, to reject your null. If a test is that fragile, then it isn't worth performing.

I'm not sure what you're interested in, but you could instead just observe how many conversions are made when recommended. In order to get a confidence interval of radius $r$, you need approximately

$$ n = \left(\dfrac{1}{r}\right)^2 $$

observations. Frankly, probably fewer than this because conversion is likely very small and so the estimated standard deviation is smaller than 0.5 (an approximation to the true standard deviation which I have made here).

All in all, don't test. Instead, estimate.