I have the results from running two algorithms and I want to be able to say that there is, say, a 95% probability that one of the sets of results is different to the other where different means A > B or B > A ("better" or "worse" in practical terms).
Basically I want to try and reject the null hypothesis that both sets of results are drawn from the same distribution in the same manner as a 2 tailed T Test or Wilcoxon test (yes I know there is a slight difference in the null hypothesis between parametric and non parametric but that's not important right now).
I want to do this with sequential sampling in which you run an initial, for example, 20 runs, carry out the test, and if there's no significant difference yet you run another run of each and repeat. The sequential sampling technique I can find the most information on is the Sequential Probability Ratio Test:
Although if people know of an alternative way of achieving the same basic goal of minimizing number of runs to prove significance that would also be helpful.
For SPRT Log L(...)/L(....) in those slides is the log likelihood ratio and my problem is I have no idea how to calculate it. It seems to be the probability of your data given the alternative hypothesis divided by the prob given the null hypothesis - but when your alternative hypothesis is just that the two data sets are different there's not enough info for you to actually calculate this probability. I'm getting the feeling that I was misled to believe SPRT was designed for this sort of hypothesis testing. So if anyone would be so kind as to either confirm that this isn't what SPRT is for or give me a concrete example for how to do this with SPRT or suggest an alternative then any info appreciated!
I should point out that in this case I am assuming that you have a rough idea of the distribution of the data (e.g. normally distributed).