0
$\begingroup$

Wondering if Fieller's theorem is used for the 'ratio' value in the compare argument in statsmodels.stats.proportion.confint_proportions_2indep method from the statsmodel module. If not, what is it? I'm trying to display the confidence interval of the % difference between p1-p2

Also, how can I interpret the numbers. I got the following 95% conf intervals with ratio: (1.05, 1.31).

EDIT: When computing for the risk ratio, the 95%CI is :(1.05, 1.31) What does it mean? Is that that the click through rate for the control group (metric being analyzed) is between 5% and 31% more likely to be clicked than the treatment group ?

Data: P1 = 30.97% n=50000, P2 = 26.34%, n=50050 Delta = -14.9% How can i get the CI for the Delta?

Improve this question
$\endgroup$
21
  • 1
    $\begingroup$ You appear to be asking two different questions. Do you need a confidence interval for the difference or for their ratio? Concerning your bold question, that's a matter of what a confidence interval means: you can find good information at stats.stackexchange.com/questions/26450. $\endgroup$
    – whuber ♦
    May 23, 2022 at 19:02
  • 1
    $\begingroup$ Re the edit and recent comments: In AB testing usually the datasets are so large you can use Normal-theory approximations. See stats.stackexchange.com/questions/388294 for instance. $\endgroup$
    – whuber ♦
    May 23, 2022 at 21:00
  • 1
    $\begingroup$ For such large datasets--as I wrote--Normal approximations will be fine. One source of software for these calculations is cran.r-project.org/web/packages/PropCIs/index.html. $\endgroup$
    – whuber ♦
    May 24, 2022 at 13:41
  • 1
    $\begingroup$ I still don't know for sure what you mean by "% difference," but for examples see stats.stackexchange.com/…. $\endgroup$
    – whuber ♦
    May 24, 2022 at 19:51
  • 1
    $\begingroup$ Okay, so at a minimum please stop calling that ratio a "difference:" you're going to confuse most readers. Call it a ratio. The Normal approximations still work. Research delta method for details. $\endgroup$
    – whuber ♦
    May 24, 2022 at 22:16

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.