0
$\begingroup$

I have two groups A and B, and for each data point I have two possible outcomes, success and failures. For each group, I have calculated the success rate, and the confidence interval for a binomial proportion. For example:

A: success rate is 27% (interval +- 5) B: success rate is 42% (interval +- 4)

I have also used the Fisher exact t-test to calculate the p-value and make sure the results are significant. My question is the following, how do I present these results? Can I say that there is a 15% improvement in success rate? with what confidence interval? what is the p-value? what else can I tell about my experiment? Thank you.

$\endgroup$
0
$\begingroup$

When comparing the means of two independent populations with different variance, assuming the sample is big enough (so we can use central limit theorem), the confidence interval for $ u_X - u_Y $ is given by:

enter image description here

where n is the number of observations from X and m is the number of observations from Y.

You can use this confidence interval interval for inference (if for a given alpha 0 is not inside the interval, then for this alpha we reject the null hypothesis), and there is also a similar formula that allows to calculate a p-value based on the same concept. There are probably functions implementing such test in all statistical programs (R, STATA, etc)

You can search the web for "confidence interval for difference of means" and you'll find many sources about this formula, a formula for the case of equal variances, etc.

$\endgroup$
  • $\begingroup$ thanks! can you please expand on what Sx and Sy are? also I have looked online for "confidence interval for difference of means" in python and I could find only applied to continuous values, like age; because my dataset is binary (success or failure) how can i apply those functions? using the central limit theorem and bootstrapping? $\endgroup$ – DarioB Feb 11 at 11:26
  • $\begingroup$ This formula is true regardless of the type of distribution, becuase of Central Limit Theorem, assuming sample size is big enough (if sample size is small, say smaller than 30, you can calculate it precisely since it is a binomial distribution). Sx^2 is the unbiased estimator for the variance of x, and Sx^4 is Sx^2 squared. $\endgroup$ – Orielno Feb 12 at 19:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.