I am testing splitting phone calls to a service between two IVR menu's to determine which messaging is more effective. 80% to the main IVR menu (Group a) and 20% to an alternative (Group b).
The data I have is Group a: Total calls, Transfer rate (10,000, 15%) and the same for group b (2000, 10%).
I want to determine whether the result is statistically significant.
So H0: No change H1: the change has been effective
As I have two proportions, should I be using a z-score? and can I assume the data is normally distributed with only these proportions?
My working so far is:
p1 = 15%, n = 10,000, sd:SQRT(0.15(1-0.15)/10,000) = 0.00357
p2 = 10%, n = 2,000, sd = 0.00671
z = (0.1-0.15)/0.00671 = -7.4535, pvalue = 3.4^-E13
Is this approach correct? If not, what should I be doing to determine whether the change has been effective?