Generally: I am interested in seeing if there is a statistically significant difference between baseline and the test conditions, given the null hypothesis that my test conditions are changing due to chance.

Specifically: I am testing different sales strategies and want to measure if the difference I'm seeing has a p-value of 0.05 or lower.

Here is an example of my data:

Trial, Success_Rate
Baseline, 0.53
Change_1, 0.41
Change_2, 0.67
Change_3, 0.93
Change_4, 0.88

I usually use the K-S statistic to generate p-values, but that is usually between cumulative distribution functions, and not with such simplified data as above. What statistical test can I perform to test significance at each change (assuming Success_Rate of Change_n is independent of Change_n-1)?


You need not just the success rate for each change (and for the baseline), $r_i/n_i$, but also the number of samples, $n_i$. Then -- assuming the counts aren't tiny -- it's a one-tailed $2\times 2$ $\chi^2$-test that you need.

Example: observed values baseline $(r_0,n_0)=(106,200)$, change 1 $(r_1,n_1)=(41,100)$. Rewrite as successes and failures, so your contingency table is

s=106  f=94 | 200    // baseline
 s=41  f=59 | 100    // change_1
s=147 f=153 | 300    // baseline + change_1 combined

So your observed and expected values are

obs 106, exp 147*200/300 : (o-e)**2/e = 0.653
obs  94, exp 153*200/300 : (o-e)**2/e = 0.627
obs  41, exp 147*100/300 : (o-e)**2/e = 1.306
obs  59, exp 153*100/300 : (o-e)**2/e = 1.255

and the $\chi^2$-statistic is the sum of those: 3.842. The right-tail $p$-value of that is about 0.05. In R:

# [1] 0.0499977

Bottom line: you need to know the weight of the evidence (the number of samples you got your 53% etc of successes from) before you can calculate the significance ($p$-value) of the change.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.