I'm interested in actually applying objective bayesian principles on real problems. In particular, I want to do sequential A/B testing with early stopping. I'm fine with assumptions of normality of the difference and known fixed variance to simplify things. This is fairly easy in the frequentist case, you just need to compute z-score stopping boundaries by simulation from a null hypothesis of 0. There is good software for this, such as the GroupSeq R package.

I've not been able to find any instances of people actually applying the objective bayesian approach to this problem though. From the theory, you can use a Jeffreys prior, which is just the non-sequential Jeffreys prior (flat in this case), multiplied by the square root of the expected stopping time. Computing the expected stopping time for any particular stopping rule is a little complex, but no more so that the simulations required by the frequentist approach.

The real complexity seems to occur if you want the stopping rule to make use of the prior, which you should if your Bayesian. You end up with a self-referential calculation, where your objective prior depends on its self for the calculation. You could do some sort of fixed point iteration I guess.

So my questions are as follows:

  • Is there a simpler sequential objective Bayes approach than what I describe above? I'm most interested in the Jeffreys or reference prior classes of priors.
  • I've read that reference priors and Jeffreys priors are the same for 1 parameter problems. Is this also true under sequential designs?
  • Are there any instances, in the literature or otherwise, of people actually using objective bayesian techniques correctly on this problem? Please provide a reference of some kind.
  • Is there any software or code out there already for solving this problem?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.