4
$\begingroup$

I have read (and will seek a specific reference on the subject) that unlike Frequentist trials, Bayesians can continually monitor data as it accrues.

A Frequentist tries to control, and thus minimize, the probability of adopting ineffective drugs. A Bayesian tries to maximize the probability that adopted drugs work. Frequentists control for multiple testing by "spending" their alpha. If one performs two or more comparisons, to preserve the overall 0.05 level of the trial, the alpha level is spent by Bonferonni, Pocock, or O'Brien Flemming formulas to control for multiple tests.

It seems, though, that when the Bayesian performs too many "interim looks" there must be some Bayes analogue to multiple testing that you are more likely to make the decision to adopt the drug on the basis of favorable interim results.

So what is the real cost to the analysis? How is it correctly reflected in a properly conducted Bayesian trial? Does the Bayesian enforce more stringent decision rules for future analyses based on the result of an interim look should we decide to keep going? Alternately, do we introduce some uncertainty in the posterior that attenuates effect somewhat to account for the fact that we might be observing a stochastic trend for an ineffective drug that may randomly show "favorable" results?

$\endgroup$
  • 2
    $\begingroup$ My understanding is that the Bayesian is subject to the same penalties as the frequentist as far as conditional error rates and so on are concerned. No escaping that. It's just that Bayesian theory doesn't care about or make any guarantees about those quantities in the first place! The things that Bayes does make guarantees about -- which is something like the proper calibration of one's beliefs -- are not hindered (and perhaps even helped) by peeking. So the Bayesian doesn't "solve" the error rate problem so much as declare it not to be a relevant problem in the first place. $\endgroup$ – Jake Westfall Jul 29 at 21:50
2
$\begingroup$

I want to be careful in answering this question, in part, because I do not ordinarily do work that requires multiple trials. There are a couple of perceptions in your post I felt should be addressed as well.

Your statement “A Frequentist tries to control, and thus minimize, the probability of adopting ineffective drugs. A Bayesian tries to maximize the probability that adopted drugs work,” may not be the best phrasing.

Consider the simple case where the null hypothesis is $\theta\le{0}$ with an alternative of $\theta>0$. There is nothing that prevents the Bayesian statistician from having two hypotheses where the first is $\theta\le{0}$, and the second is $\theta>0$. They do not have the same meaning, however.

I was reminded of this one day when my wife came home from assisting in a childbirth. The hospital she was at was one of two hospitals. It delivered high-risk births, and so the weights of the babies were barbelled. They either tended to be very low or very high. The other hospital tended to have the center of the bell curve.

A baby that was over thirteen pounds was delivered that day, with the other babies being tiny. I asked her the weights, and I realized through quick mental calculation, that I could not falsify the null hypothesis that the babies’ mass was less than or equal to zero using a Frequentist method, but I had a roughly ninety-five percent chance that the parameter was greater than zero using a Bayesian method. Ignoring the preposterousness and the lack of random selection, it illustrates the difference in meaning. So you are correct, Frequentist and Bayesian hypothesis tests may not mean the same thing, but it may not be safe to assume that one is performing a minimization while the other is performing its dual optimization.

As to interim looks, there are two potential answers to the question. I do want to note, though, that it is possible to perform “looks” with a Frequentist method if one simply chooses to be careful. It is true that stopping problems were noted as a problem for Frequentist methods, but that just lead to a revision of Frequentist methods.

Meta-analysis, for example, allows the combining of multiple studies. Control methods, such as those for family-wise errors, have been developed. It is true, nonetheless, that such a tool is far easier to construct in the Bayesian sphere than the Frequentist.

There are two cases to consider. The first case is related to the replication crisis in the sciences, the second to manufacturing. The solutions would not naturally be the same.

One of the causes of the replication crisis is that only statistically significant studies tend to be published. The implication is that published parameters tend to be biased away from no effect, though the set of all observed parameters may have no systematic bias at all. In the case of “peeking,” it will be true that the posterior will never forget a long random run that happened early in the process. Furthermore, because Bayesian methods do not forget, the parameter estimate will be biased systematically, in the bad sense of bias, if and only if studies are stopped due to non-significance and chosen due to significance. Remember that Bayesian methods are generally biased methods regardless.

I would point out that statistical significance is not a Bayesian concept. However, there is an analog of something having a high posterior probability. Still, as mentioned above in the baby example, it may not mean the same thing as the Frequentist inference.

Frequentist decisions minimize the risk of the worst outcome; Bayesian decisions minimize the average loss. Risk is a function; loss is a scalar. They are not always doing the same thing even when the calculations are the same. As a consequence of that fact, plus the fact that the Bayesian model is looking at the probability of the model given the data, runs can be a bit more consequential for the Bayesian making a decision in the extreme cases.

Nonetheless, it is informative to know that parameters may be biased significantly away from zero because only those studies selected based on the posterior will be continued. So let us consider study one with posterior $\pi(\mu|\sigma;X)$ where $\mu$ is the center of location, $\sigma$ is the scale parameter, and $X$ is the data. Let us imagine that the posterior is normal with the posterior mean of twenty-five, and the variance is one.

The posterior would not be altered until we came to a decision node. At a decision node, if the study were continued, then the posterior would be calculated, but it would not be used as a prior in the follow-on elements of the study because it may be upwardly biased.

While the posterior to the right of twenty-five should be untouched, the posterior to the left may contain more mass than is warranted by the posterior as you know you are ignoring some research paths. One could place a flat mass over the region to the left of twenty-five where a Frequentist would just barely reject the observed value. One could then continue the left side of the normal distribution at that point.

In effect, you forget information in the data, but you are substituting the information in your professional judgment for the information in the data before you start your next research leg.

You could also perform a large scale validation process to determine if you have a stable false positive rate. In that case, you could use that data to influence the next prior.

The danger in this, the case of the run, is that you will bias your parameters downward by imposing a flat prior on the region to the left of the observed posterior mean (or right had the test results been on the other side of the axis). That is a conservative choice that may cause you to reject valid treatments in the following way.

Imagine that in the first study, you saw a truly representative sample of data and the drug’s effect was strong and very helpful to the participants. Based on conservativeness, the posterior is flattened to be the prior of the next study, which, unfortunately, resulted in no effects due to an unfortunate choice of sample. While the drug would have been approved had the prior not been tampered with, the drug is rejected due to the second peek having been bad.

That is the danger in altering decision rules in a Bayesian space based on peeking. The assumption is that the second peek is good, and the first is bad. It may be the other way around.

Note that, depending a bit on the assumptions, that the Frequentist meta-analysis may have accepted the treatment because it wouldn’t flatten the prior to protect from bias as it is already using an unbiased estimator.

There is a conservatism in Frequentist decisions that are lost in Bayesian methods unless they are specifically built into the process. In addition to the use of a flat prior in the region approaching zero, a minimax decision rule may be available even in Bayesian methodologies. That would also rebuild the conservatism in a Bayesian space.

However, there is a second case that needs to be considered, one where posterior does not influence continuing the research. In that case, the posterior should not be touched in any way because it is not biasing which research projects are continued forward. Parallel assembly lines often have this property. Peeking doesn’t alter the measurements because peeking doesn’t cause the process to stop forever, though it may cause a temporary stop. In the case of quality assurance, it is desirable to know the population parameters and peeking is valuable. Indeed, if the posterior probability will not influence the stopping of a research project, then peeking does not cause harm, and no adjustments should be made. The Bayesian posterior probability automatically adjusts itself for the multiple look problem. There is no Bayesian analog to the Holm-Bonferroni test as it is unnecessary.

One last parting thought. The idea of type 1 and type 2 error is not a Bayesian construction and can fall apart when the number of hypotheses is higher than two. There is no null hypothesis. Still, in the binary case, you could discuss rejecting useful versus useless treatments. The difficulty is that Bayesian methods do not have long-run Frequentist properties. There is no way to assure power and no way to guarantee a level of protection against false positives. That is because Bayesian methods work in the parameter space.

While there is no concept of power in Bayesian experiments, one could ask what sample size would be required “to change my mind,” to a certain level of certainty. Likewise, there is no guarantee against false positives, but simulations can be constructed under a “null” on the performance of a Bayesian method over a simulated sample space. It will not give you a guarantee, but it will give you a feel for performance issues under Frequentist ideals.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.