13
$\begingroup$

Ie, to do sequential analysis (you don't know ahead of time exactly how much data you will collect) with frequentist methods requires special care; you can't just collect data until the p-value gets sufficiently small or a confidence interval becomes sufficiently short.

But when doing Bayesian analysis, is this a concern? Can we freely do things like collect data until a credible interval becomes sufficiently small?

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Depends. If you collect data until you have a certain amount of information that is generally not an issue, whether you are Bayesian or frequentist. If you care about frequentist operating characteristics (e.g. coverage probabilities for credible intervals, type 1 error), it is still an issue to stop e.g. once the credible interval excludes no effect. $\endgroup$
    – Björn
    Commented Dec 28, 2015 at 18:43
  • $\begingroup$ @Björn Can you explain what "a certain amount of information" means in this context? And even if we don't get constant type 1 error rates with sequential bayesian testing, are we still "allowed" to? Can we still safely make the usual claims made in a Bayesian analysis? (ie, statements about the probability distribution of a parameter) $\endgroup$
    – alecbz
    Commented Dec 28, 2015 at 18:53
  • 1
    $\begingroup$ Certain amount of information = e.g. Fisher information (e.g. for survival analysis up to a certain number of cases). For the second question: yes, if you use the likelihood reflecting how you sampled (i.e. e.g. reflecting, in which cases you would have stopped collecting more data). No, if you ignore what the correct likelihood is (and e.g. just use a standard normal likelihood ). $\endgroup$
    – Björn
    Commented Dec 28, 2015 at 20:25
  • $\begingroup$ Ah, I see now, so the problem is really in the likelihood I guess. A stopping rule makes future observations conditionally dependent on prior ones. $\endgroup$
    – alecbz
    Commented Dec 28, 2015 at 22:49
  • $\begingroup$ @Bjorn Do you know of any reference for a Bayesian analysis that takes into account a stopping rule in its liklihood function? $\endgroup$
    – alecbz
    Commented Dec 29, 2015 at 2:19

2 Answers 2

Reset to default
6
$\begingroup$

Rouder (2014) has a nice paper on this (written for psychologists), explaining why sequential testing (so-called data peeking) is fine from a Bayesian perspective. (Paper is freely available online if you do a search for it.)

Schoenbrodt et al. (in press) present nice analyses showing how to use sequential analysis with Bayes factors to determine when to stop data collection.

From a Bayesian parameter estimation procedure, John Kruschke has a very nice blog post that compares different Bayesian methods during sequential testing.

Hope you find them of help.

References

Rouder, Jeffrey N. (2014). Optional stopping: No problem for Bayesians. Psychonomic Bulletin & Review, 21, 301-308.

Schoenbrodt, F. D., Wagenmakers, E.-J., Zehetleitner, M., & Perugini, M. (in press). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Could you summarize the papers instead of providing just the citations? $\endgroup$
    – Tim
    Commented Jun 27, 2019 at 10:34
0
$\begingroup$

SPRT is good example of a frequentist method that is sequential.

On the other hand, while Bayesian models have priors to overcome sparsity of data, the more data you have the "narrower" your posterior distribution becomes making it less suitable for online temporal learning.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.