4
$\begingroup$

Since the frequentist's p-values are uniformly distributed under the null hypothesis, it is a highly problematic practice to add more and more data to your sample until you find a significant result. Assuming the null hypothesis is true, my understanding is that this will almost assuredly lead to a Type I error. This is bad scientific practice.

However, I often hear that Bayesian statistics does not suffer the same fate. Is this true?

If little evidence for the alternative hypothesis exists for some given sample size, wouldn't only stopping once there is "sufficient" evidence for the alternative hypothesis also be problematic for the Bayesian?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ See sequential analysis. $\endgroup$
    – Glen_b
    Dec 9, 2018 at 6:19
  • $\begingroup$ I have since come across an article by Rouder (2014; doi: 10.3758/s13423-014-0595-4). He demonstrates quite convincingly that observed posterior odds (even with optional stopping) are representative of the truth. $\endgroup$
    – NBland
    Dec 10, 2018 at 4:17

1 Answer 1

Reset to default
4
$\begingroup$

It's not that the procedure you describe (keep collecting data until you like the results) does not inflate the type 1 error, if you naively conduct repeated Bayesian analyses, it's that the brand of Bayesian that considers that there is no issue in repeatedly looking at data simply considers type 1 errors an irrelevant concept - and would likely also not favor looking at whether a credible interval excludes 0 to make a decision.

An alternative way of looking at this is to write down the likelihood based on the whole experiment - e.g. if I can never see more heads than tails, because I will keep flipping coins until I see more tails than heads, then the final outcome of the experiment clearly does not follow a binomial distribution.

Another typical way tho handle multiplicity in a Bayesian setting is to use a hierarchical model, but I have never seen a clear description of how one would do that in this context.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Let's say we do a Bayesian analysis to see whether an unbiased coin is fair (the alternate hypothesis being that the probability of heads on any given toss is not equal to 50%). There will be sequences of tosses that appear biased, even though they are due to chance. But if we stop tossing the coin when this discrepancy between observed heads and observed tails approaches what we deem significant, then that is clearly a Type I error...right? I suppose it's difficult to intuit why Bayesian statistics with optional stopping doesn't lead to false positive evidence like a frequentist approach. $\endgroup$
    – NBland
    Dec 9, 2018 at 9:09
  • 1
    $\begingroup$ In case the answer was not clear: of course it does lead to a higher type I error rate, but the argument for why to ignore it is that we should not care about the type 1 error rate. $\endgroup$
    – Björn
    Dec 9, 2018 at 9:13
  • 1
    $\begingroup$ Shouldn't we always care about falsely positive evidence? The Bayesian and the frequentist share the goal of good scientific practice, and Type I errors are counter to this goal. $\endgroup$
    – NBland
    Dec 9, 2018 at 9:16
  • 1
    $\begingroup$ @NBland: Your last comment is very interesting and merits being stated as its own question. Maybe frequentist and bayes concerns with different aspect of inference? maybe frequency and subjective (more or less) opinion is different aspect of probability, not just diferent interpretations? $\endgroup$
    – kjetil b halvorsen ♦
    Dec 9, 2018 at 9:31
  • $\begingroup$ Perhaps you would rather control the false discovery rate? If you only study true null hypotheses 100% of your claimed discoveries will be false. I have a lot of sympathy for type 1 error control, but it's not the be all end all of science. $\endgroup$
    – Björn
    Dec 9, 2018 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.