Question about the Bayesian Inference of a parameter

Question

In order to understand the difference between the Frequentist and Bayesian inference, I was reading the presentation at: http://www.stat.ufl.edu/archived/casella/Talks/BayesRefresher.pdf . In order to explain the difference between the approaches, the author is using the following example(Page 6): If I understand correctly, there are three different cases of protocols (something related to toxicity, it seems) and there are different studies for each of the three protocols, which has examined the relation of the protocol to a specific cause, which is called something as "AAN".

For example, for one of the three protocols, different studies showing the total number of cases (first) and relation to "AAN" (second) is as following :

1) 66,11

2) 1756,129

3) 272, 48

4) 151, 18

... etc. Each of these number pairs belong to different studies.

Now, the Bayesian model for these studies are given as such:

Given $X_j$ is the AAN frequency of the $j$th study about the protocol $i$, $X_j$ is distributed as $X_j \sim Binomial(n_j,p_i)$. $n_j$ is the total number of incidences for the $j$th study and $p_i$ is the parameter of the distribution we want to infer.

What I did not understand here is, the author says that, in this model $p_i$ can vary from study to study.

In my understanding of the Bayesian approach, here, all of the studies constitute our data, $D=\left( n_1,X_1,n_2,X_2,...,n_K,X_K \right)$ where $K$ is the total number of studies. For $p_i$ we have a prior distribution of $P(p_i)$. So we try to find the posterior distribution $P(p_i|D)$. In my understanding $p_i$ cannot vary from study to study: $p_i$ is first generated from the prior distribution as $p_i \sim P(p_i)$ and then this generated value of $p_i$ is used to generate each $X_j \sim Binomial(n_j,p_i)$. Yes, $p_i$ is not fixed as in the frequentist approach, but it varies over different realizations of all $K$ studies, once a $p_i$ is generated from the prior, then all $K$ studies use the same $p_i$. So, it should not change from a single study to another. So, this slide has confused me at this particular point. Am I right with my thoughts here or did I misunderstand something?

Thanks in advance.

Answer 1

I think the slides are a bit ambiguous. For the Bayesian approach you can (among many other things) assume that:

1. All protocols share the same $p$. Then there's a single prior over what that $p$ might be.

2. Each study $i$ has its own $p_i$. Then you can either

   2.1. Assume that there's nothing in common to the three protocols, so that nothing in the results of one protocol could be informative about the others. Then you might want three independent priors, one for each $p_i$. Or, perhaps more reasonably...

   2.2 Assume that all are these three protocols are a 'sample' of the possible protocols that could have been run. In which case it's natural to think of the $p_i$ for each protocol to be a draw from a common prior over $p$. You might formalise this as reflecting an exchangeability assumption, but that doens't much matter for your question.

Assumption 1 (plus model and data) will generate a marginal posterior over $p$. Both of assumptions 2 will generate a marginal posterior over [$p_1, p_2, p_3$], but with potentially different properties.

But I suppose what the slides are going for is the idea that, from a Frequentist perspective, a natural null hypothesis to test is that $p=p_1=p_2=p_3$. Rejecting this suggests that there is variation across the $p_i$ which one might investigate further.

However one might, in equally Frequentist mode, simply ask for confidence intervals for the different $p_i$ and see if they overlap (assuming you could get the coverage right, etc.) This would give results rather closer to those generated through assumption 2.1., without formally assuming, even as a null hypothesis, that there was a single fixed $p$.

This second approach makes the Frequentist and Bayesian approaches look much more like each other. And spoils the overly sharp contrast that the slides seems to be making.