I have the results from running two algorithms and I want to be able to say that there is, say, a 95% probability that one of the sets of results is different to the other where different means A > B or B > A ("better" or "worse" in practical terms).

Basically I want to try and reject the null hypothesis that both sets of results are drawn from the same distribution in the same manner as a 2 tailed T Test or Wilcoxon test (yes I know there is a slight difference in the null hypothesis between parametric and non parametric but that's not important right now).

I want to do this with sequential sampling in which you run an initial, for example, 20 runs, carry out the test, and if there's no significant difference yet you run another run of each and repeat. The sequential sampling technique I can find the most information on is the Sequential Probability Ratio Test:

http://www.stats.ox.ac.uk/~steffen/teaching/bs2siMT04/si14c.pdf http://en.wikipedia.org/wiki/Sequential_probability_ratio_test

Although if people know of an alternative way of achieving the same basic goal of minimizing number of runs to prove significance that would also be helpful.

For SPRT Log L(...)/L(....) in those slides is the log likelihood ratio and my problem is I have no idea how to calculate it. It seems to be the probability of your data given the alternative hypothesis divided by the prob given the null hypothesis - but when your alternative hypothesis is just that the two data sets are different there's not enough info for you to actually calculate this probability. I'm getting the feeling that I was misled to believe SPRT was designed for this sort of hypothesis testing. So if anyone would be so kind as to either confirm that this isn't what SPRT is for or give me a concrete example for how to do this with SPRT or suggest an alternative then any info appreciated!

I should point out that in this case I am assuming that you have a rough idea of the distribution of the data (e.g. normally distributed).

Many thanks!

  • 2
    $\begingroup$ For Binomial data, formulas and a simple worked example are provided by William Q. Meeker Jr., A Conditional Sequential Test for the Equality of Two Binomial Proportions. Appl. Stat. (1981) 30, No. 2, pp 109-115 (available at JSTOR). I recently applied this to a Web "AB test" involving very long sequences; it performed as claimed and was reasonably efficient to compute. $\endgroup$ – whuber Apr 21 '15 at 15:54

You can use some hierarchical Bayes model to model the corresponding null and alternative distribution. For the prior distributions, you could use empirical Bayes type methods to approximate from data or use uninformative priors. Then you can use EM algorithm to update the parameters.

| cite | improve this answer | |
  • 1
    $\begingroup$ I think the OP put the problem in the context of sequential analysis. I don't see how your suggestion addresses that. $\endgroup$ – Michael R. Chernick Mar 30 '18 at 21:23
  • $\begingroup$ Please refer to Wei and Hero’s paper in 2013, 2014. They have a detailed description on how to implement in sequential analysis. $\endgroup$ – Jack Mar 31 '18 at 22:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.