Examples of Bayesian and frequentist approach giving different answers

Question

Note: I am aware of philosophical differences between Bayesian and frequentist statistics.

For example "what is the probability that the coin on the table is heads" doesn't make sense in frequentist statistics, since it has either already landed heads or tails -- there is nothing probabilistic about it. So the question has no answer in frequentist terms.

But such a difference is specifically not the kind of difference I'm asking about.

Rather, I would like to know how their predictions for well-formed questions actually differ in the real world, excluding any theoretical/philosophical differences such as the example I mentioned above.

So in other words:

What's an example of a question, answerable in both frequentist and Bayesian statistics, whose answer is different between the two?

(e.g. Perhaps one of them answers "1/2" to a particular question, and the other answers "2/3".)

Are there any such differences?

• If so, what are some examples?

• If not, then when does it actually ever make a difference whether I use Bayesian or frequentist statistics when solving a particular problem?
  Why would I avoid one in favor of the other?

Answer 1

This example is taken from here. (I even think I got this link from SO, but cannot find it anymore.)

A coin has been tossed $n=14$ times, coming up heads $k=10$ times. If it is to be tossed twice more, would you bet on two heads? Assume you do not get to see the result of the first toss before the second toss (and also independently conditional on $\theta$), so that you cannot update your opinion on $\theta$ in between the two throws.

By independence, $$f(y_{f,1}=\text{heads},y_{f,2}=\text{heads}|\theta)=f(y_{f,1}=\text{heads})f(y_{f,2}=\text{heads}|\theta)=\theta^2.$$ Then, the predictive distribution given a $\text{Beta}(\alpha_0,\beta_0)$-prior, becomes \begin{eqnarray*} f(y_{f,1}=\text{heads},y_{f,2}=\text{heads}|y)&=&\int f(y_{f,1}=\text{heads},y_{f,2}=\text{heads}|\theta)\pi(\theta|y)d\theta\notag\\ &=&\frac{\Gamma\left(\alpha _{0}+\beta_{0}+n\right)}{\Gamma\left(\alpha_{0}+k\right)\Gamma\left(\beta_{0}+n-k\right)}\int \theta^2\theta ^{\alpha _{0}+k-1}\left( 1-\theta \right) ^{\beta _{0}+n-k-1}d\theta\notag\\ &=&\frac{\Gamma\left(\alpha_{0}+\beta_{0}+n\right)}{\Gamma\left(\alpha_{0}+k\right)\Gamma\left(\beta_{0}+n-k\right)}\frac{\Gamma\left(\alpha_{0}+k+2\right)\Gamma\left(\beta_{0}+n-k\right)}{\Gamma\left(\alpha_{0}+\beta_{0}+n+2\right)}\notag\\ &=&\frac{(\alpha_{0}+k)\cdot(\alpha_{0}+k+1)}{(\alpha_{0}+\beta_{0}+n)\cdot(\alpha_{0}+\beta_{0}+n+1)} \end{eqnarray*} For a uniform prior (a $\text{Beta}(1, 1)$-prior), this gives roughly .485. Hence, you would likely not bet. Based on the MLE 10/14, you would calculate a probability of two heads of $(10/14)^2\approx.51$, such that betting would make sense.

Answer 2

See my question here, which mentions a paper by Edwin Jaynes that gives an example of a correctly constructed frequentist confidence interval, where there is sufficient information in the sample to know for certain that the true value of the statistic lies nowhere in the confidence interval (and thus the confidence interval is different from the Bayesian credible interval). For a simpler example (after MacKay) see my answer to the related question "Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?".

However, the reason for this is the difference in the definition of a confidence interval and a credible interval, which in turn is a direct consequence of the difference in frequentist and Bayesian definitions of probability. If you ask a Bayesian to produce a Bayesian confidence (rather than credible) interval, then I suspect that there will always be a prior for which the intervals will be the same, so the differences are down to choice of prior.

Whether frequentist or Bayesian methods are appropriate depends on the question you want to pose, and at the end of the day it is the difference in philosophies that decides the answer (provided that the computational and analytic effort required is not a consideration).

Being somewhat tongue in cheek, it could be argued that a long run frequency is a perfectly reasonable way of determining the relative plausibility of a proposition, in which case frequentist statistics is a slightly odd subset of subjective Bayesianism - so any question a frequentist can answer a subjectivist Bayesian can also answer in the same way, or in some other way should they choose different priors. ;o)

Answer 3

I believe this paper provides a more purposeful sense of the trade-offs in actual applications between the two. Part of this might be due to my preference for intervals rather than tests.

Gustafson, P. and Greenland, S. (2009). Interval Estimation for Messy Observational Data. Statistical Science 24: 328–342.

With regard to intervals, it may be worthwhile to keep in mind that frequentist confidence intervals require/demand uniform coverage (exactly or at least great than x% for each and every parameter value that does not have zero probability) and if they don't have that - they arn't really confidence intervals. (Some would go further and say that they must also rule out relevant subsets that change the coverage.)

Bayesian coverage is usually defined by relaxing that to "on average coverage" given the assumed prior turns out to be exactly correct. Gustafson and Greenland (2009) call these omnipotent priors and consider falliable ones to provide a better assessment.

Answer 4

If someone were to pose a question that has both a frequentist and Bayesian answer, I suspect that someone else would be able to identify an ambiguity in the question, thus making it not "well formed".

In other words, if you need a frequentist answer, use frequentist methods. If you need a Bayesian answer, use Bayesian methods. If you don't know which you need, then you may not have defined the question unambiguously.

However, in the real world there are often several different ways to define a problem or ask a question. Sometimes it is not clear which of those ways is preferable. This is especially common when one's client is statistically naive. Other times one question is much more difficult to answer than another. In those cases one often goes with the easiest while trying to make sure his clients agree with precisely what question he is asking or what problem he is solving.

Answer 5

I recommend looking at Exercise 3.15 of the freely-available textbook Information Theory, Inference and Learning Algorithms by MacKay.

When spun on edge 250 times, a Belgian one-euro coin came up heads 140 times and tails 110. 'It looks very suspicious to me', said Barry Blight, a statistics lecturer at the London School of Economics. `If the coin were unbiased the chance of getting a result as extreme as that would be less than 7%'. But do these data give evidence that the coin is biased rather than fair?

The example is worked out in detail on pp. 63-64 of the textbook. The conclusion is that the $p$-value is $0.07$, but the Bayesian approach gives varying levels of support for either hypothesis, depending on the prior. This ranges from a recommended answer of no evidence that the coin is biased (when a flat prior is used) to an answer of no more than $6:1$ against the null hypothesis of unbiasedness, in the case that an artificially extreme prior is used.

Answer 6

A funny buth insightfull example is given by xkcd in https://xkcd.com/1132/:

It stands for a whole group of problems where we have a strong prior and Frequentism neglects the prior. The Frequentist compares how likely the result is in the light of the null hypothesis but she does not consider whether the hypothesis is a priori even much more unlikely.

So they both come to opposite conclusions.

Answer 7

An important area where the two approaches will yield conflicting assessments is the context of multiplicity. Since p-values involve the probability of getting more extreme results than the results observed if a null hypothesis is true, having more looks at the data will increase the p-value. For Bayes on the other hand, more looks at the data just result in more rapid updating of evidence, and previous evidence assessments are now obsolete and can be completely ignored. Bayesian measures are study time-respecting while frequentist $\alpha$ probability is non-directional. Two classes of examples are (1) sequential testing where frequentist approaches are well developed but are conservative and (2) situations in which there is no way to use a frequentist approach to even address the problem of interest.

• In a sequential study, using Bayes one may look at the data infinitely often without changing the definition or reliability of posterior probabilities. The frequentist approach becomes increasingly conservative as the number of looks increases.
• In a study in which there are multiple endpoints, the frequentist approach has a great deal of difficulty even putting together an overall evidentiary measure, while the Bayesian approach has no difficulty. For example suppose that one is developing a migraine headache drug and the outcomes are sleep problems, pain, nausea, light sensitivity, and sound sensitivity. One may reasonably claim the drug to be a success if there is a high posterior probability that the drug improved any 3 of the 5 patient outcomes. The only frequentist methods that have been proposed are closed testing procedures that seek evidence for any or all of the 5 endpoints being benefited by drug.

Another major category where Bayes disagrees with frequentist is the frequent case where the frequentist result in incorrect from a frequentist standpoint. This occurs quite generally when the log likelihood has a very non-Gaussian shape, for example with binary logistic regression with an imbalanced Y. While the uncommonly used profile likelihood interval yields fairly accurate confidence interval coverage probabilities, the most commonly used approaches such as the Wald method and various bootstrap intervals do not. You will see inaccurate tail non-coverage probabilities in at least one of the two tails. Bayesian highest posterior density or credible intervals are exact on the other hand, for all sample sizes.

Answer 8

The answer provided by Christoph Hanck compares a Bayesian prediction interval for a future experimental result with a frequentist point estimate of a population-level parameter. A more appropriate comparison would be to compare a Bayesian prediction interval with a frequentist prediction interval. In my examples below I compare Bayesian posterior intervals with frequentist confidence intervals.

Ultimately, the choice of using a Bayesian or frequentist approach comes down to how you choose to define probability. See my post here on interpretation and why one would choose a frequentist interpretation, Bayesian vs frequentist interpretations of probability. The following is taken from my manuscript on confidence distributions - Johnson, Geoffrey S. "Decision Making in Drug Development via Confidence Distributions" Researchgate.net (2021). In short, objective Bayesian and frequentist inference will differ the most when the data distribution is skewed, the sample size is small, and inference is performed near the boundary of the parameter space. Below I will begin with an example where Bayesian and frequentist inference agree perfectly. I will then provide another example where they differ.

Under $H_0$: $\theta=\theta_0$ the likelihood ratio test statistic -2log$\lambda(\boldsymbol{X},\theta_0)$ follows an asymptotic $\chi^2_1$ distribution (Wilks 1938). If an upper-tailed test is inverted for all values of $\theta$ in the parameter space, the resulting distribution function of one-sided p-values is called a confidence distribution function. That is, the one-sided p-value testing $H_0$: $\theta\le\theta_0$, \begin{eqnarray}\label{eq} H(\theta_0,\boldsymbol{x})= \left\{ \begin{array}{cc} \big[1-F_{\chi^2_1}\big(-2\text{log}\lambda(\boldsymbol{x},\theta_0)\big)\big]/2 & \text{if } \theta_0 \le \hat{\theta}_{mle} \\ & \\ \big[1+F_{\chi^2_1}\big(-2\text{log}\lambda(\boldsymbol{x},\theta_0)\big)\big]/2 & \text{if } \theta_0 > \hat{\theta}_{mle}, \end{array} \right. \end{eqnarray} as a function of $\theta_0$ and the observed data $\boldsymbol{x}$ is the corresponding confidence distribution function, where $\hat{\theta}_{mle}$ is the maximum likelihood estimate of $\theta$ and $F_{\chi^2_1}(\cdot)$ is the cumulative distribution function of a $\chi^2_1$ random variable. Typically the naught subscript is dropped and $\boldsymbol{x}$ is suppressed to emphasize that $H(\theta)$ is a function over the entire parameter space. This recipe of viewing the p-value as a function of $\theta$ given the data produces a confidence distribution function for any hypothesis test. The confidence distribution can also be depicted by its density defined as $h(\theta)=dH(\theta)/d\theta$.

Consider the setting where $X_1,...,X_n\sim\text{Exp}(\theta)$ with likelihood function $L(\theta)=\theta^{-n} e^{-\sum{x_i}/\theta}$. Then $supL(\theta)$ yields $\hat{\theta}_{mle}=\bar{x}$ as the maximum likelihood estimate for $\theta$, the likelihood ratio test statistic is $-2\text{log}\lambda({\boldsymbol{x},\theta_0})\equiv-2\text{log}\big(L({\theta}_0)/L(\hat{\theta}_{mle}) \big)$, and the corresponding confidence distribution function is defined as above. The histogram above, supported by $\bar{x}$, depicts the plug-in estimated sampling distribution for the maximum likelihood estimator (MLE) of the mean for exponentially distributed data with $n=5$ and $\hat{\theta}_{mle}=1.5$. Replacing the unknown fixed true $\theta$ with $\hat{\theta}_{mle}=1.5$, this displays the estimated sampling behavior of the MLE for all other replicated experiments, a $\text{Gamma}(5,0.3)$ distribution. The Bayesian posterior depicted by the thin blue curve resulting from a vague conjugate prior or an improper 1/θ prior is a transformation of the likelihood and is supported on the parameter space, an $\text{Inverse Gamma}(5,7.5)$ distribution. The bold black curve is also data dependent and supported on the parameter space, but represents confidence intervals of all levels from inverting the likelihood ratio test. It is a transformation of the sampling behavior of the test statistic under the null onto the parameter space, a ``distribution" of p-values. Each value of $\theta$ takes its turn playing the role of null hypothesis and hypothesis testing (akin to proof by contradiction) is used to infer the unknown fixed true $\theta$. The area under this curve to the right of the reference line is the p-value or significance level when testing the hypothesis $H_0$: $\theta \ge 2.35$. This probability forms the level of confidence that $\theta$ is greater than or equal to 2.35. Similarly, the area to the left of the reference line is the p-value when testing the hypothesis $H_0$: $\theta \le 2.35$. One can also identify the two-sided equal-tailed $100(1-\alpha)\%$ confidence interval by finding the complement of those values of $\theta$ in each tail with $\alpha$/2 significance. The dotted curve shows the exact likelihood ratio confidence density formed by noting that $\bar{X}\sim$ Gamma$(n,\theta/n)$ and inverting its cumulative distribution function. This confidence density agrees perfectly with the posterior distribution. A confidence density similar to that based on the likelihood ratio test can be produced by inverting a Wald test with a log link. When a normalized likelihood approaches a normal distribution with increasing sample size, Bayesian and frequentist inference are asymptotically equivalent.

In the example above the posterior mean agrees with the maximum likelihood estimate. This is not always the case. Take, for example, estimation and inference on a non-linear monotonic transformation of $\theta$.

For an example where Bayesian and frequentist inference differ, consider the setting where $X_1,...,X_n\sim\text{Bernoulli}(\theta)$ with likelihood function $L(\theta)=\theta^{\sum x_i}(1-\theta)^{n-\sum x_i}$. The conjugate Bayesian posterior is a $\text{Beta}(a+\sum x_i, b+n-\sum x_i)$ where $a$ and $b$ are the prior parameters. If a vague conjugate prior is used and $19$ events are witnessed in a sample of size $n=20$, the Bayesian posterior becomes $\text{Beta}(a+19, b+20-19)$. This produces a posterior mean point estimate of $\frac{a+19}{a+19+b+20-19}$. Below are two posterior density estimates with 95% credible intervals based on vague conjugate priors, one with $a=1$ and $b=1$, and another with $a=0.1$ and $b=0.1$. Also plotted are confidence curves, one-sided p-values calculated using the cumulative distribution function for $\sum X_i\sim$ $\text{Bin}(n=20,\theta)$, as well as the resulting 95% confidence interval. The Bayesian and frequentist point and interval estimates are similar but different. I suspect this is due at least in part to the parameter space being continuous and the sample space for $\sum X_i$ being discrete.

In terms of a willingness to bet, the Bayesian bets according to his beliefs while the frequentist bets according to the long-run probability of his testing procedure. This betting is best imagined in terms of a betting market. The question becomes, what would the market decide on, personal belief or long-run performance? Most gamblers would agree that long-run performance is the best bet.

If there is no relevant historical data the frequentist would be willing to bet $\$0.95$ and expect $\$1$ in return if his $95\%$ confidence interval contains the true $\theta$ based on the long-run characteristics of the test above, whereas the Bayesian would be willing to bet more than $\$0.95$ for the same interval based on his beliefs that all $\theta$'s were equally likely (or concentrated near 0 and 1) until $19$ events were witnessed in a sample of $n=20$. The frequentist would gladly "buy this bet in the market" at $\$0.95$ and "sell it" (play the bookie) to the Bayesian to make a risk-free profit regarless of whether the frequentist or Bayesian interval covers the true $\theta$. To the frequentist at no point was $\theta$ randomly selected from a $\text{Beta}(a, b)$ distribution and then imagined instead to have been seleted from a $\text{Beta}(a+\sum x_i, b+n-\sum x_i)$.

The Bayesian prior represents subjective belief. It can also be used to incorporate historical data. Under the frequentist paradigm, historical data can be incorporated via a fixed-effect meta-analysis. (1)

Answer 9

The answer provided by Christoph Hanck compares a Bayesian predictive probability to a frequentist point estimate. Below are frequentist p-values for making a prediction (Johnson 2021a) as compared to a Bayesian posterior predictive distribution and a discussion on willingness to bet. Recall the problem statement,

"A coin has been tossed $n=14$ times, coming up heads $k=10$ times. If it is to be tossed twice more, would you bet on two heads? Assume you do not get to see the result of the first toss before the second toss (and also independently conditional on $\theta$), so that you cannot update your opinion on $\theta$ in between the two throws."

Let $\theta$ be the probability of heads and $r$ be the number of heads in the next two flips. To predict the result of two coin tosses the frequentist will calculate the p-value testing the hypothesis that the next two throws will be heads, $H_0: r=2$ or equivalently $H_0: \hat{\theta}_2=1$, using an ancillary quantity where $\hat{\theta}_2=r/2$ and $\hat{\theta}_{14}=k/14$. A candidate for such a quantity could involve

$$\hat{\theta}_{14}-\hat{\theta}_2. $$

Based on 100,000 simulations the sampling distribution for this quantity is approximately ancillary, meaning it approximately does not depend on the unknown fixed true $\theta$. Near $\theta=10/14=0.71$ we could try approximating it with a bell curve.

Since our observed estimate and hypothesized value yield $\hat{\theta}_{14}-\hat{\theta}_2 = 10/14 - 2/2 = -0.29$, the lower-tailed p-value using an approximate Wald test would be

$$\Phi\Bigg(\frac{\hat{\theta}_{14}-\hat{\theta}_{2}}{\hat{\text{se}}}\Bigg)=0.20, $$

where $\hat{\text{se}}=\sqrt{\hat{\theta}_{14}(1-\hat{\theta}_{14})/14 + \hat{\theta}_{14}(1-\hat{\theta}_{14})/2}$. If we increase the number of bins in our histogram and look specifically at the sampling distribution of our statistic at $\theta=10/14$ we see that it may not look quite so bell shaped.

Referencing this sampling distribution the lower-tailed probability of $\hat{\theta}_{14}-\hat{\theta}_{2}=-0.29$ or something more extreme is $0.31$, not far off from our crude Wald p-value. Below is a table of predictive p-values as a function of the hypothesis for $r$ being tested based on the above simulated sampling distribution, as well as posterior predictive belief "probabilities" based on a $\text{Beta}(1,1)$ prior, $k=10$, and $n=14$. There are in fact two conclusions the frequentist could draw based on the observed data and the approximate sampling distribution. Had we witnessed, say, $k=9$ or $k=11$ this ambiguity would not have occurred. Interestingly, the Bayesian finds himself in a similar predicament.

	$$H_0:\hat{\theta}_2=0\iff H_0: r=0$$	$$H_0:\hat{\theta}_2 = 0.5\iff H_0: r = 1$$	$$H_0:\hat{\theta}_2=1\iff H_0: r=2$$
Predictive p-value [smallest one-sided]	0.05 [upper]	0.34 [upper]	0.31 [lower]
Confidence curve 1 [one-sided p-value]	0.05 [upper]	0.34$|$0.76 [upper$|$lower] (0.64)	0.31 [lower]
Confidence curve 2 [one-sided p-value]	0.05 [upper]	0.34 [upper]	0.81 [upper] (0.66)
Posterior predictive distribution	0.11	0.40	0.49

1. Conclusion 1

   To the frequentist, $H_0: r=1$ or equivalently $H_0: \hat{\theta}_2=0.5$ could be regarded as the most plausible hypothesis because it has the largest tail probability of $0.34$ and the other two hypotheses have smaller p-values. This is consistent with the point estimate $\hat{\theta}_{14}=0.71$ since this estimate is closer to $H_0: \hat{\theta}_2=0.5$ than to $H_0: \hat{\theta}_2=1$. Not only can $\hat{\theta}_{14}=0.71$ be viewed as an estimate for the unknown fixed true $\theta$, it can also be viewed as an estimate for the as of yet unobserved $\hat{\theta}_2$. To the Bayesian, $r=2$ has the largest predictive posterior belief and is therefore most "probable" (plausible). This is contrary to the conclusion drawn by Christoph Hanck that the Bayesian would not bet on $r=2$ while the frequentist would. To the frequentist, since we can "rule out" $r=0$ at the $0.05$ level and $r=2$ at the $0.31$ level we are $64\%$ confident in the alternative which is $r=1$. This confidence level is nothing more than a restatement of the p-values for $r=0$ and $r=2$, $100(1-0.05-0.31)\%$. In the long run $\hspace{0mm}64\%$ of the time we see a discrepancy like $\hat{\theta}_{14}-\hat{\theta}_2=10/14-1/2=0.21$ [i.e. within (-0.29, 0.71)] regardless of the unknown fixed true $\theta$ (approximately). These statements of confidence are presented in the table above as "Confidence curve 1." Based on the observed data the frequentist would be most surprised by a result of $r=0$ followed by $r=2$, and least surprised by $r=1$.

   In terms of a willingness to bet the frequentist would bet $\$0.31$ and expect $\$1$ in return if $r=2$ based on the long-run characteristics of the test above, whereas the Bayesian would be willing to bet $\$0.49$ based on his beliefs that all $\theta$'s were equally likely until the coin came up heads $k=10$ times in $n=14$ throws. The frequentist would gladly "buy this bet in the market" at $\$0.31$ and "sell it" (play the bookie) to the Bayesian at $\$0.49$ making a risk-free $\$0.49-\$0.31=\$0.18$ regardless of whether the coin actually lands twice on heads (Dutch book). To the frequentist at no point was the coin randomly selected from a uniform distribution and then imagined instead to have been selected from a $\text{Beta}(1+10,1+14-10)$.

2. Conclusion 2

   Since the upper-tailed p-value testing $H_0:r= 1$ is smaller than the sum of the p-values testing $H_0:r=0$ and $H_0:r=2$, the frequentist could conclude that $H_0:r=2$ is the most plausible hypothesis. That is, since we can rule out $H_0:r=1$ (and by extension $H_0:r=0$) at the one-sided $0.34$ level we are therefore $66\%$ confident in the alternative which is $r=2$. This is shown in Confidence curve 2 (note these p-values are not expected to sum to 1). In the long run $66\%$ of the time we see a discrepancy like $\hat{\theta}_{14}-\hat{\theta}_2=10/14-2/2=-0.29$ [i.e. greater than 0.21] regardless of the unknown fixed true $\theta$ (approximately). While this does not coincide with the point estimate $\hat{\theta}_{14}=0.71$ being closest to $H_0:\hat{\theta}_2=0.5$, it does coincide with the probability mass of the sampling distribution being concentrated most heavily around $\hat{\theta}_{14}-\hat{\theta}_2$ $=10/14-2/2$ $=-0.29$. Based on this conclusion the frequentist would be most surprised by a result of $r \le 1$ and least surprised by $r=2$. To the Bayesian, $r=0$ and $r=1$ have a combined posterior predictive belief of $0.51$, and therefore $r\ne 2$ is most "probable" (plausible).

   In terms of a willingness to bet the frequentist would bet $\$0.66$ and expect $\$1$ in return if $r=2$ based on the long-run characteristics of the test above, whereas the Bayesian would be willing to bet $\$0.49$ based on his beliefs that all $\theta$'s were equally likely until the coin came up heads $k=10$ times in $n=14$ throws. The frequentist would gladly "buy the other side in the market" at $\$1-\$0.66=\$0.34$ and "sell it" (play the bookie) to the Bayesian at $\$1-\$0.49=\$0.51$ making a risk-free $\$0.51-\$0.34=\$0.17$ regardless of whether the coin actually lands twice on heads (Dutch book).

The second conclusion could be deemed the most appropriate since it results in a higher confidence level for the favored outcome. This interpretation utilized evidential p-values under a Fisherian framework. Under a Neyman-Pearson framework one would pre-specify a single strawman null hypothesis or default prediction, say, $H_0:r=2$ and pre-specify an arbitrary type I error rate, say, $\alpha=0.35$. Based on the results above, this hypothesis would be rejected in favor of the alternative, $H_a:r\ne 2$. If, however, one had pre-specified a lower type I error rate of, say, $\alpha=0.20$ the strawman default prediction of $H_0:r=2$ would have been retained. This approach is useful if there are no consequences when predicting $H_0:r=2$ and getting it wrong, and dire consequences when predicting $H_a:r\ne 2$ and getting it wrong.

Below are confidence curves for $\theta$ from inverting an exact likelihood ratio test based on our sample of size $n=14$, which produces a 95% confidence interval for $\theta$ of (0.42, 0.92). Since our quantity $\hat{\theta}_{14}-\hat{\theta}_2$ for predicting $r$ is not exactly ancillary we could perform a number of sensitivity analyses to see how the p-value changes depending on the unknown fixed true $\theta$. The above analysis and suggested sensitivity analysis are less than ideal since the quantity we chose is not exactly ancillary (the sampling distribution is not exactly the same for any value of $\theta$). We could look for another quantity that is exactly ancillary or a closer approximation to ancillary.

Many Bayesians would presume as Christoph Hanck did that the frequentist is forced to assume that $\theta$ is known to be exactly equal to $10/14$ and reference the above binomial sampling distribution for $r$ when making predictions. This might seem at odds with our prediction above. After all, our best estimate for $\theta$ is $\hat{\theta}_{14}=0.71$ and therefore our best estimate for the probability of observing two heads, $\theta^2$, is $\hat{\theta}_{14}^2=0.51$. However, by chance alone we could have easily observed $9$ out of $14$ heads leading to an estimate of $\hat{\theta}_{14}^2=0.41$. The confidence curves below from inverting an exact likelihood ratio test account for the sampling variability of our estimator with a $64\%$ confidence interval matching the level of confidence in our prediction for $r$. Based on the long-run probability of our experiment we are $5\%$ confident that $\theta^2$ is less than or equal to $0.21$, $31\%$ confident that $\theta^2$ is greater than or equal to $0.64$, and therefore $64\%$ confident that $\theta^2$ lies within $(0.21, 0.64)$. If $\theta^2$ is truly $0.21$ then the probability that $r=1$ is $0.50$, which is in line with conclusion 1. However, if $\theta^2$ is near $0.64$ then $r=2$ has the highest probability, in line with conclusion 2. Performing inference on $\theta^2$ has the benefit of exact inference on a continuous hypothesis space compared to forming a prediction for $r$. It also formulates the solution in terms of a direct probability statement about $r$ while accounting for the uncertainty around having estimated $\theta$. This may make it easier for non-statisticians to interpret and understand (Johnson 2021 b). Nevertheless, this solution though exact still leaves ambiguity as to which bet to take.

The real solution to the problem might simply be that we do not have enough information to reliably predict $r$, whether constructing predictive p-values or performing inference on $\theta^2$. For more information we would need to increase our sample size $n$ (Johnson 2021 c). Although the Bayesian and frequentist results differ numerically, a similar conclusion is reached under both paradigms since a Bayesian belief of $0.50$ is interpreted as "undecided." While the Bayesian can reliably model his beliefs for any sample size, beliefs are not facts. If our prediction is not based on facts, it will not be reliable. No one is ever interested in how a prophet believes in his prediction. They only ever care about how often he gets it right.

%let k=10;
%let n=14;

data sim;
*do theta=0.1, 0.25, 0.5, 0.75, 0.9;
do theta=round(&k./&n.,0.01);
do sim=1 to 100000;
    y14=rand('binomial',theta,&n.);
    y2=rand('binomial',theta,2);

    *T=log( ((y2/2)/(1-y2/2)) / ((y14/&n.)/(1-y14/&n.))  );

    *T=log( (y2/2)/(y14/&n.) );

    *T=(y2/2)/(y14/&n.);

    T=y14/&n.-y2/2;

    output;
end;
end;
run;


%let Wald_r0=;
%let Wald_r1=;
%let Wald_r2=;

data pvalue;
do r=0 to 2 by 1;
theta14=&k./&n.;
theta2=r/2;
t=theta14-theta2;
se=sqrt( theta14*(1-theta14)/&n. + theta14*(1-theta14)/2 );
Wald_pvalue=min(cdf('normal',t/se ,0 ,1), 1-cdf('normal',t/se ,0 ,1));

if r=2 then  call symput('Wald_r2',Wald_pvalue);
if r=1 then  call symput('Wald_r1',Wald_pvalue);
if r=0 then  call symput('Wald_r0', Wald_pvalue);
output;
end;
run;

data pvalue;
set pvalue;


if r=2 then Wald_confidence=1-&Wald_r0.-&Wald_r1.;
if r=1 then Wald_confidence=1-&Wald_r0.-&Wald_r2.;
if r=0 then Wald_confidence=1-&Wald_r2.-&Wald_r1.;

max=max(&Wald_r0.,&Wald_r1.,&Wald_r2.);

if round(Wald_pvalue,0.001) ne round(max,0.001) then Wald_confidence=Wald_pvalue;

keep r Wald_pvalue Wald_confidence;
run;

proc print data=pvalue noobs;
var r Wald_pvalue Wald_confidence;
footnote 'One-sided Wald p-values';
run;
footnote;

ods escapechar="^";
ods graphics / height=3in width=6in border=no;
proc sgpanel data=sim;
panelby theta / onepanel rows=1;
histogram T ;*/ nbins=6;
*density T / type=normal;
refline %sysevalf(&k./&n.-1) / axis=x lineattrs=(color=darkblue thickness=2);
refline %sysevalf(&k./&n.-0) / axis=x lineattrs=(color=darkblue thickness=2);
refline %sysevalf(&k./&n.-0.5) / axis=x lineattrs=(color=darkblue thickness=2);
colaxis label="^{unicode hat}^{unicode theta}14 - ^{unicode hat}^{unicode theta}2";
run;


proc sort data=sim out=sort;
by T;
run;

ods trace on;
ods select none;
proc univariate data=sim pctldef=3;
var t;
cdf t;
ods output cdfplot=cdf;
run;
ods select all;

data cdf;
set cdf;
ecdfx=round(ecdfx,0.0001);
run;

data cdf;
set cdf;
by ecdfx;
if not last.ecdfx then delete;
complementary_cdf=100-ecdfy;
complementary_cdf=lag(complementary_cdf);
if complementary_cdf=. then complementary_cdf=100;
if ecdfx=round(&k./&n.-1,0.0001) then r=2;
if ecdfx=round(&k./&n.-0.5,0.0001) then r=1;
if ecdfx=round(&k./&n.-0,0.0001) then r=0;
if r ne . and ecdfy lt complementary_cdf then refline1=ecdfx;
if r ne . and complementary_cdf lt ecdfy then refline2=ecdfx;
run;

proc sgplot data=cdf;
refline  refline1/ axis=x;
step x=ecdfx y=ecdfy / markers markerattrs=(symbol=circlefilled);
title 'p-value from simulated sampling distribution';
xaxis  grid minor minorgrid label="^{unicode hat}^{unicode theta}14-^{unicode hat}^{unicode theta}2";
yaxis grid minor minorgrid label="Cumulative distribution function";
run;
title; footnote;

proc sgplot data=cdf;
refline  refline2 / axis=x;
step x=ecdfx y=complementary_cdf / markers markerattrs=(symbol=circlefilled);
title 'p-value from simulated sampling distribution';
xaxis grid minor minorgrid label="^{unicode hat}^{unicode theta}14-^{unicode hat}^{unicode theta}2";
yaxis grid minor minorgrid label="Complementary CDF";
run;
title; footnote;

proc sort data=cdf out=cdf_pvalue (where=(r ne .)); by r; run;


data cdf_pvalue;
set cdf_pvalue;
pvalue=min(ecdfy, complementary_cdf);

if r=2 then call symput('r2', pvalue); 
if r=1 then call symput('r1', pvalue);
if r=0 then call symput('r0', pvalue);

run;

data cdf_pvalue;
set cdf_pvalue;

p0=&r0.; p1=&r1.; p2=&r2.;

if r=2 then confidence=100-p0-p1;
if r=1 then confidence=100-p0-p2;
if r=0 then confidence=100-p2-p1;



max=max(p0, p1, p2);

if round(pvalue,0.001) ne round(max,0.001) then confidence=pvalue;

pvalue2=pvalue/100;
confidence2=confidence/100;

*keep r pvalue2 confidence2;
rename pvalue2=pvalue confidence2=confidence;
run;


proc print data=cdf_pvalue noobs;
var r pvalue confidence;
footnote 'One-sided p-values from simulated sampling distribution';
run;
footnote;





data bayes;
do sim=1 to 100000;
    theta=rand('beta',1+&k.,1+&n.-&k.);
    r=rand('binomail',theta,2);
    output;
end;
run;

ods select none;
proc univariate data=bayes;
var r;
cdf r;
ods output cdfplot=cdf_bayes;
run;
ods select all;

ods noproctitle;
proc freq data=bayes;
table r /  nocum;
footnote 'Bayesian predictive distribution';
run;
footnote;

Answer 10

There are four sources of differences between Bayesian, Likelihoodist, Frequentist, and the various uncategorized miscellaneous methods, such as the method of moments, that I have been able to discover. The first has to with differences in defining the idea of an optimal solution. The second has to do with the deeper layers of the methods, such as the axiom structure. The third has to do with prior knowledge. The fourth has to do with the intent of the model.

My daughter and I were home some very many years ago, and neither my wife nor sons were home. My guess is that they were involved in a band activity or maybe lacrosse. We had cake the day before, and when I opened the refrigerator, I realized that there was too much cake for one person but not enough cake for two people. I asked her if she wanted to split the last piece. I grabbed a knife to cut it, and she said, "can I cut the cake?"

I said, "sure, but if you cut, then I get to choose the piece." That seemed most amenable to her little girl's mind, and I could see the wheels turning as she moved the knife around to work out how uneven she should cut the cake and not raise any protest. She finally made her cut, and I grabbed my piece of cake.

A loud protest rang out, and I offered to put the pieces back together again to try and make the cut better. Her bewildered look was followed by an emphatic "no," from her. So I shrugged, walked off, and ate the cake leaving her with the small piece.

It was then that she realized her father was related to Darth Vader.

My daughter had used her prior knowledge of when I cut the cake and let her pick, had assigned a zero prior probability to my statement, and chose her fair cut. The likelihood, when combined with her posterior, changed her view of many parameters.

This mentality of "I cut the cake, and you choose" is the basis of one of the axiomatizations of probability. I have just shown you its potential weakness. You should always assume a being of utter darkness is on the other side of the deal. My daughter now understands why I cast a shadow even when there is no direct source of light.

Let us assume that I will let you set the gambling odds, but I will choose what combination of bets to take. Presumably, you will not accept a gamble where I am certain to win regardless of the outcome of the event being gambled on. You won't play, "heads, I win; tails, you lose."

So we are going to play a cake-cutting game and then gamble on it. By gambling on it, we change the intention of the game. We also invoke a very particular axiom set that we can contrast with another set. In addition, we can ponder the idea of optimality. Tangentially, we can discuss the small impact prior knowledge has.

By making you the financial intermediary, I am giving you control of the "bid" and the "ask," or alternatively, the vig, using bookies' terminology. To make the game fair, we agree that I will pay a flat fee per cut cake. In return, I can bet any finite sum. I can either go long or short on any gamble as well. We will assume that I have adequate collateral to cover any offered short bet.

Also, to make the game fair, you agree to use the risk-neutral measure as you are collecting a flat fee. In other words, you will grant fair odds. The reason for the flat fee is that lump sums are constants, and so their derivative is zero. The fee does not play a factor in profit optimization.

You have two tasks. The first is to cut the cake, the second to give prices for lotteries. Either the left side or the right side of the cake will be larger.

In the simple games of economics without uncertainty, you would want the cake to be cut as follows in the illustration. You want it to be cut in such a way that you cannot distinguish which side is bigger.

Unfortunately, this is not a game of perfect knowledge. We need to play a statistical game that has uncertainty in it, or it isn't any fun.

To add uncertainty, we will have a robot put the cake in a darkened room so that neither can see the cake's location. A point on the table will be chosen by a random number generator. The generator will produce uniformly distributed points.

A robot will enter the room and place the center of this perfectly circular cake on the randomly chosen point. A sensor will detect points from the cake in a uniformly distributed manner. In other words, both you and I will be provided with forty sets of points. Each point on the cake is equiprobable to be detected by the sensor.

You will then cut the cake through the minimum variance unbiased estimator (MVUE) of the center using an ultrapowerful laser cutter.

Because the MVUE is guaranteed by force of math to be perfectly accurate, half the time the left side will be larger, as the sample size tends to infinity. If you granted even odds, and I always bet on the left side, one would expect a binomial distribution of outcomes over thirty cakes to look thusly.

But would betting on left every time be a rational decision on my part?

After all, I have data. I could form a posterior distribution for the parameter and do another estimate. Since I have a computer, I could do it before you cut the cake, based only on our shared forty points. What if my Bayesian estimate didn't agree with your Frequentist estimate?

Now, this is where a bunch of mathematical tricks comes in. The intent is to gamble, which changes the outcome. If we were not gambling, although there would be differences, they wouldn't really matter. Indeed, all that would matter would be how we defined optimality upfront, determining which solution we should choose.

Gambling is different because Frequentist statistics give rise to arbitrage opportunities at least in some percentage of the repetitions of a game.

The issue has to do with a conflict between the Dutch Book Theorem and Kolmogorov's axioms. The appearance of this conflict varies from game to game but is always present. It sometimes takes a bit of digging to find the conflict, but it will be there as a pathology. There also has to be an opponent. Someone has to know that there is a mispricing of a lottery such as a stock, horse racing ticket, or options on wheat futures.

The specific reason is that Kolmogorov's third axiom is that the union of partitions of sets yield countably additive measures. You can cut a continuous probability distribution into a countably infinite number of parts under Frequentist axioms. The Dutch Book Theorem says that you cannot do that. You can cut sets, but the number of sets must be finite.

My objective function would be to win the largest number of gambles. It would not care if I used an unbiased method unless that maximizes wins.

Now the prior does give me a slight edge. A cake near the border will, in part, sit off the table. Points over the edge cannot sit in a location that could also be the center of the cake. If all the observed points came from places over the edge, even though that would be an exceedingly rare combination of events, it would allow me to know that you have a mistake in your calculations. If the MVUE were sitting in midair instead of on the table, I would know your calculations were wrong.

Nonetheless, we have forty data points. If we expanded the table enough, we could make that case vanishingly slight.

Still, there is a problem. I have data from which I can construct a posterior density. Bayesian bets cannot be arbitraged under mild additional conditions. Lotteries priced with Frequentist tools can be arbitraged at least some of the time.

The Frequentist cake cut would pass through the circle like this. Please note that the circle is an oval on some machines due to differences in devices and settings.

However, the MVUE is not inside the posterior. You can know that because you can place a circle of equal radius around the MVUE, and you will get the following graph.

Obviously, the MVUE sits in a physically impossible location. That permits anybody to know which side will be larger. Knowing the points is sufficient to detect the differences between the two types of models against the reality.

The green lines map the Bayesian posterior. The black point minimizes quadratic loss, but each point in the posterior is an equally valid possible location for the center of the cake.

I did a simulation putting a corner at (10,10) and (90,90), and I was guaranteed a win 48% of the time. Additionally, I won more than half the remaining gambles due to differences between the methods.

For the MVUE to be guaranteed to be unbiased, it has to pay for that insurance policy with information. That difference in information results in a difference in precision.

Because you are cutting at an angle from (0,0), you must be outside the joint Cartesian posterior and the marginal distribution of the posterior angle or slope. The MVUE can be closer to the true center even though it is outside the Cartesian posterior.

If I used your cuts and your odds for thirty rounds, my expected value, assuming I bet 100% of my money every round I knew the correct answer and made the Kelly-optimal bet the remaining 52% of the time, would be to make 128,000 times my initial amount. Under the Frequentist model where I always bet left or a random location, I would expect a small loss due to the fee I was paying to play.

I have a half dozen of these games related to asset pricing, trading, or options pricing.

Real-world financial models are complicated because if you attempt to derive the Frequentist models in a Bayesian framework, you do not end up with models that look alike at all. It doesn't help, necessarily, to cut and paste a Bayesian procedure on an economic model built on top of Frequentist axioms.

Nonetheless, you can learn quite a bit when you cut a cake with a child involved.

Answer 11

A very obvious case where it makes a difference is when there is a relevant prior information, but we are trying to analyze a small dataset. An example that I found quite useful (and used in my thesis, full details of the analysis below are given there) is the TGN1412 first in human trial.

During that first-in-human trial of a monoclonal antibody (i.e. TGN1412), all 6 of 6 the healthy volunteers in the test group ended up in a critical care unit due to cytokine storms, while this occurred for 0 of the 2 placebo subjects.

Adverse event	TGN1412 (N=6)	Placebo (N=2)
Patient in ICU due to Cytokine storm	6/6 (100%)	0/2 (0%)

As Stephen Senn pointed out, Fisher's exact test results in a one-sided p-value of 0.0357 (i.e. above 0.025). Also, if you do an exponential time-to-event model using exact Poisson regression, you get a median unbiased estimate of 1.05 (95% CI -0.62 to $\infty$) for the log-hazard ratio with a two-sided p-value of 0.3308. So, two somewhat reasonable frequentist analyses would normally be interpreted as (ignoring that this analysis was not pre-specified etc.) having not enough data to reject the null hypothesis that the drug increases the likelihood of the cytokine storms. Nevertheless, when people talk about this trial, you will not here any doubt that the drug caused these adverse events. Why is that?

Let's use historical discharges or deaths from intensive care units from around then (2001, the trial was in 2006) for the UK (both trial and historical data). That's probably an upper bound for the expected rate, because this is the general population including more frail individuals (rather than the young and healthy volunteers in the trial) and the ICU stays were for any case rather than for cytokine storms. From that I get a Gamma(1270614; 4808670538) prior for the control group exponential rate per patient year (probably reasonable to use an exponential distribution). If I take a Cauchy(0, 0.25) prior for the log-hazard ratio of TGN1412 vs. placebo, then I get a posterior median log-hazard ratio of 12.2 (95% CrI 11.3 to 13.0) with a posterior probability in excess of 99.999% that TGN1412 increased the hazard rate for the admission to critical care.

I.e. the prior knowledge on the rarity of the specic adverse event (a cytokine storm requiring admission to an intensive care unit) in young and healthy individuals means that these adverse events have been attributed to the TGN1412, because we a-priori would expect to see zero such cases with very high probability in such a short small trial. The fact that we saw 6 cases in the TGN1412 out of 6 patients is so implausible unless the drug caused it, that people are very convinced of a causal drug effect in this case.

Other examples were Bayesian methods make a huge difference is when your data tells you very little (or essentially nothing) about certain parameters in your model, but when there is prior information on these parameters. Especially when we do not know the exact value of the parameters, a Bayesian treatment often becomes important. E.g. a lot of phyisis constants such as the speed of light are known with so little error that for many purposes it makes very little difference whether we take a Bayesian approach that accounts for the uncertainty around them or whether plug in a constant. However, other quantities we know a decent amount about, but still have a non-negligible amount of uncertainty about. In those situations a Bayesian approach is a good way of propagating the uncertainty into an analysis.