# What can be learnt from the sampling distribution of posterior probabilities?

I have an rather open question. In Bayesian statistics you do testing based on some posterior distribution $p(\theta|D)$. E.g. you could try something like $T=P[\theta>0|D]$ and decide based on the threshold $.5$.

$T$ is basically some statistics of the data observed - and as such has a sampling distribution. Could I not get frequentists bounds on $T$ that guarantee a certain error niveau using the null? Am I completely mistaken here, is it just useless or does this approach have a tag I could not find?

The cool thing would be having a frequentist test proceedure and instead of messing around with 'almost' significant p-values you just report the statistic $T$ itself? Is this related to conditional testing?

There's some problems with the way you're asking the question, as far as I can tell (or perhaps i'm misreading your question).

So, frequentist hypothesis testing says given that the null hypothesis $H_0: \theta \in \Theta_0$ is true (versus $H_1: \theta \in \Theta_1$), reject $H_0$ if the test statistic $T = \frac{\sup_{\theta \in \Theta_1} p(D | \theta)}{\sup_{\theta \in \Theta_0} p(D | \theta)}$ is sufficiently large. Notice that at no point in this is there any indication that $\theta$ is a random variable. The unknown value $\theta$ is considered fixed and unknown. Thus a null hypothesis of $H_0: \theta = 0$ (i.e a single-valued hypothesis) is a reasonable hypothesis to pose.

In Bayesian hypothesis testing, there is no notion of assuming $H_0$ to be true - just a prior distribution on $\theta$ that weights the various hypotheses. You base your acceptance/rejection criterion on a ratio depending on $p(\theta|H_0)$ and $p(\theta|H_1)$. Notice that $\theta$ is has the posterior distribution associated to it, and so a statement like $H_0:\theta = 0$ will have posterior probability of 0 if the posterior for $\theta$ is a continuous distribution, and so in such cases, it doesn't make sense to have such a null hypothesis - rather you desire to ask $H_0: \theta \in \Theta_0$ for $\Theta_0$ being small interval containing 0, or something like that. Note that the statements $p(\theta|H_0)$ and $p(\theta|H_1)$ are completely meaningless in frequentist testing.

Note also that posterior probabilities in no way correspond to p-values - the posterior probabilities are probabilities on $\theta$ given the fixed data $D$ while p-values ask about probabilities of the random data $D$ given fixed $\theta$. These two approaches ask fundamentally different questions. The former asks what you should believe about $\theta$ based on the observed data (and only the observed data), and the latter asks whether your data is extreme assuming the null hypothesis is true (i.e. it considers hypothetical data).

Your question asks whether you can get some sort of frequentist bounds on the posterior probabilities given that $H_0$ is true. I suppose you can ask how the posterior probabilities would vary if the data were to vary according to the null hypothesis, as a measure of sensitivity of the posterior to the data. But this wouldn't answer the original hypothesis testing question of whether to accept/reject the hypotheses based on the available data.

I hope I haven't misinterpreted your question and I hope this even makes sense and is correct...

## Posterior probabilities as statistics

$T$ is some statistics of the data observed - and as such has a sampling distribution.

This is actually not the case. $T = P \left(\theta > 0 \mid \text{data}\right)$ as you have defined it is just a posterior probability. It is, strictly speaking, not a statistic. That is, it is not a measure of the data. It does change with the data observed, but it does not have a sampling distribution in the classical sense.

Here's another way to look at the issue. Given that you believe in your likelihood & prior specification, your uncertainty about $\left( \theta \mid \text{data} \right)$ is completely characterized by the posterior distribution. The more data you collect, the more confident the posterior of $\left( \theta \mid \text{data} \right)$ will be by a mathematically appropriate amount.

For an example, consider a beta-binomial model. $y \sim \text{Binomial}(n, \phi)$ where $\phi = 0.8$ secretly, but we give the prior $\phi \sim \text{Beta}(1,1) = \text{Uniform}(0,1)$. We can simulate $y$ for different $n$ (which generalizes to the size of the data) and watch our posterior distribution on $\left( \phi \mid \text{data} \right)$. Here are some horizontal-scale-constant plots for $n = 10, 50, 1000$.

As you can see, our belief about $\phi$ given the data changes dramatically in shape with the data size. This corresponds to the natural idea that the data is changing our ideas about $\phi$ in such a way that we grow more and more confident about what $\phi$ is with more and more data. With this particular model the posterior variance decreases with the data size. The useful idea of looking at a high-probability range of $\phi \mid \text{data}$ is analogous (and in special cases mathematically equivalent to) the confidence interval under the frequentist paradigm. You might want to look at Bayesian credible intervals. Bayesian credible intervals have Frequentist coverage probabilities which are often very close to the Bayesian coverage level. This book (which I found relatively approachable) has a small discussion on the topic and can point you towards academic sources.

## Testing with posterior probabilities

The cool thing would be having a frequentist test proceedure and instead of messing around with 'almost' significant p-values you just report the statistic $T$ itself?

You certainly could report $T$, as that would be your posterior belief that $\theta > 0$. Since the Bayesian paradigm doesn't take into account hypothesis preference (i.e., favor the null), when making decisions it is useful to consider the cost of a particular decision. For example, a particular super market chain may have a posterior belief that a certain kind of cereal will sell worse this upcoming season, but they wouldn't want to remove this cereal completely from stock unless their posterior belief were high enough. They can minimize their risk by assigning different costs to each decision (e.g., accept/reject the null). For example., you might prefer the null hypothesis 7 times as much as the alternative. This lecture is an excellent primer on this particular aspect of Bayesian inference/decision-making (and actually brings up a way to deal with the null-hypothesis-is-a-point concern of queenbee).

Bayes factors are another notable model selection approach which can be used to assess the relative merits of different models in the Bayesian paradigm. They have been compared to Frequentist p-values in the sense that they can quantify the weight of evidence on each side of the hypothesis tug-of-war.