# compute p-value for test conditions vs baseline conditions

Generally: I am interested in seeing if there is a statistically significant difference between baseline and the test conditions, given the null hypothesis that my test conditions are changing due to chance.

Specifically: I am testing different sales strategies and want to measure if the difference I'm seeing has a p-value of 0.05 or lower.

Here is an example of my data:

Trial, Success_Rate
Baseline, 0.53
Change_1, 0.41
Change_2, 0.67
Change_3, 0.93
Change_4, 0.88


I usually use the K-S statistic to generate p-values, but that is usually between cumulative distribution functions, and not with such simplified data as above. What statistical test can I perform to test significance at each change (assuming Success_Rate of Change_n is independent of Change_n-1)?

You need not just the success rate for each change (and for the baseline), $r_i/n_i$, but also the number of samples, $n_i$. Then -- assuming the counts aren't tiny -- it's a one-tailed $2\times 2$ $\chi^2$-test that you need.

Example: observed values baseline $(r_0,n_0)=(106,200)$, change 1 $(r_1,n_1)=(41,100)$. Rewrite as successes and failures, so your contingency table is

s=106  f=94 | 200    // baseline
s=41  f=59 | 100    // change_1
------------+----
s=147 f=153 | 300    // baseline + change_1 combined


So your observed and expected values are

obs 106, exp 147*200/300 : (o-e)**2/e = 0.653
obs  94, exp 153*200/300 : (o-e)**2/e = 0.627
obs  41, exp 147*100/300 : (o-e)**2/e = 1.306
obs  59, exp 153*100/300 : (o-e)**2/e = 1.255


and the $\chi^2$-statistic is the sum of those: 3.842. The right-tail $p$-value of that is about 0.05. In R:

pchisq(3.841536,1,lower.tail=F)
#  0.0499977


Bottom line: you need to know the weight of the evidence (the number of samples you got your 53% etc of successes from) before you can calculate the significance ($p$-value) of the change.