0
$\begingroup$

I am testing splitting phone calls to a service between two IVR menu's to determine which messaging is more effective. 80% to the main IVR menu (Group a) and 20% to an alternative (Group b).

The data I have is Group a: Total calls, Transfer rate (10,000, 15%) and the same for group b (2000, 10%).

I want to determine whether the result is statistically significant.

So H0: No change H1: the change has been effective

As I have two proportions, should I be using a z-score? and can I assume the data is normally distributed with only these proportions?

My working so far is:

p1 = 15%, n = 10,000, sd:SQRT(0.15(1-0.15)/10,000) = 0.00357

p2 = 10%, n = 2,000, sd = 0.00671

z = (0.1-0.15)/0.00671 = -7.4535, pvalue = 3.4^-E13

Is this approach correct? If not, what should I be doing to determine whether the change has been effective?

Thanks

$\endgroup$
  • $\begingroup$ "Group a: Total calls, Transfer rate (10,000, 15%) and the same for group b (2000, 10%). I want to determine whether the result is statistically significant. " ... if by "result" you mean to ask whether the proportions are more different than could be accounted for by random variation, plainly they are. That much difference with that large a sample size? of course the population proportions are different (unless your significance level is extremely tiny, I guess). Effectiveness would seem to at least be in part a matter of effect size, rather than significance -- I'd focus on tat $\endgroup$ – Glen_b -Reinstate Monica Jun 2 '17 at 6:56
0
$\begingroup$

Your Z value is wrong (the denominator should be the square root of the pooled variance), but it hardly matters ... if by "result" you mean to ask whether the proportions are more different than could be accounted for by random variation, plainly they are. That much difference with that large a sample size? of course the population proportions are different! What would be the purpose in testing such a plain difference?

Effectiveness would seem to at least be in part a matter of effect size, rather than significance -- I'd focus on that (though the size seems like it would be practically important as well).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.