How do Bayesians interpret $P(X=x|\theta=c)$, and does this pose a challenge when interpreting the posterior?

Question

I have seen the post Bayesian vs frequentist interpretations of probability and others like it but this does not address the question I am posing. These other posts provide interpretations related to prior and posterior probabilities, $\pi(\theta)$ and $\pi(\theta|\boldsymbol{x})$, not $P(X=x|\theta=c)$. I am not interested in the likelihood as a function of the parameter and the observed data, I am interested in the interpretation of the probability distribution of unrealized data points.

For example, let $X_1,...,X_n\sim Bernoulli(\theta)$ be the result of $n$ coin tosses and $\theta\sim Beta(a,b)$ so that $\pi(\theta|\boldsymbol{x})$ is the pdf of a $Beta(a+\sum x,b + n - \sum x)$.

How do Bayesians interpret $\theta=c$? $\theta$ of course is treated as an unrealized or unobservable realization of a random variable, but that still does not define or interpret the probability of heads. $\pi(\theta)$ is typically considered as the prior belief of the experimenter regarding $\theta$, but what is $\theta=c$? That is, how do we interpret a single value in the support of $\pi(\theta)$? Is it a long-run probability? Is it a belief? How does this influence our interpretation of the prior and posterior?

For instance, if $\theta=c$ and equivalently $P(X=1|\theta=c)=c$ is my belief that the coin will land heads, then $\pi(\theta)$ is my belief about my belief, and in some sense so too is the prior predictive distribution $P(X=1)=\int\theta\pi(\theta)d\theta=\frac{a}{a+b}$. To say "if $\theta=c$ is known" is to say that I know my own beliefs. To say "if $\theta$ is unknown" is to say I only have a belief about my beliefs. How do we justify interpreting beliefs about beliefs as applicable to the coin under investigation?

If $\theta=c$ and equivalently $P(X=1|\theta=c)=c$ is the unknown fixed true long-run probability for the coin under investigation: How do we justify blending two interpretations of probability in Bayes theorem as if they are equivalent? How does Bayes theorem not imply there is only one type of probability? How are we able to apply posterior probability statements to the unknown fixed true $\theta=c$ under investigation?

The answer must address these specific questions. While references are much appreciated, the answers to these questions must be provided. I have provided four Options or proposals in my own solution below as an answer, with the challenges of interpreting $P(X=x|\theta=c)$ as a belief or as a long-run frequency. Please identify which Option in my answer most closely maps to your answer, and provide suggestions for improving my answer.

I am not writing $P(X=x|\theta=c)$ to be contemptuous. I am writing it to be explicit since $P(X=x|Y=y)$ is not the same thing as $P(X=x|Y)$. One might instead be inclined to write in terms of a sample from the prior and use an index of realizations of $\theta$. However, I do not want to present this in terms of a finite sample from the prior.

More generally, how do Bayesians interpret $P(X=x|\theta=c)$ or $P(X\le x|\theta=c)$ for any probability model and does this interpretation pose any challenges when interpreting $P(\theta=s|\boldsymbol{x})$ or $P(\theta\le s|\boldsymbol{x})$?

I've seen a few other posts tackle questions about Bayesian posterior probability, but the solutions aren't very satisfying and usually only consider a superficial interpretation, e.g. coherent representations of information.

Related threads:

Examples of Bayesian and frequentist approaches giving different results

Bayesian vs frequentist interpretations of probability

UPDATE: I received several answers. It appears that a belief interpretation for $P(X=x|\theta=c)$ is the most appropriate under the Bayesian paradigm, with $\theta$ as the limiting proportion of heads (which is not a probability) and $\pi(\theta)$ representing belief about $\theta$. I have amended Option 1 in my answer to accurately reflect two different belief interpretations for $P(X=x|\theta=c)$. I have also suggested how Bayes theorem can produce reasonable point and interval estimates for $\theta$ despite these shortcoming regarding interpretation.

Answer 1

I have posted a related (but broader) question and answer here which may shed some more light on this matter, giving the full context of the model setup for a Bayesian IID model.

You can find a good primer on the Bayesian interpretation of these types of models in Bernardo and Smith (1994), and you can find a more detailed discussion of these particular interpretive issues in O'Neill (2009). A starting point for the operational meaning of the parameter $\theta$ is obtained from the strong law of large numbers, which in this context says that:

$$\mathbb{P} \Bigg( \lim_{n \rightarrow \infty} \frac{1}{n} \sum_{i=1}^n X_i = \theta \Bigg) = 1.$$

This gets us part-way to a full interpretation of the parameter, since it shows almost sure equivalence with the Cesàro limit of the observable sequence. Unfortunately, the Cesàro limit in this probability statement does not always exist (though it exists almost surely within the IID model). Consequently, using the approach set out in O'Neill (2009), you can consider $\theta$ to be the Banach limit of the sequence $X_1,X_2,X_3$, which always exists and is equivalent to the Cesàro limit when the latter exists. So, we have the following useful parameter interpretation as an operationally defined function of the observable sequence.

Definition: The parameter $\theta$ is the Banach limit of the sequence $\mathbf{X} = (X_1,X_2,X_3,...)$.

(Alternative definitions that define the parameter by reference to an underlying sigma-field can also be used; these are essentially just different ways to do the same thing.) This interpretation means that the parameter is a function of the observable sequence, so once that sequence is given the parameter is fixed. Consequently, it is not accurate to say that $\theta$ is "unrealised" --- if the sequence is well-defined then $\theta$ must have a value, albeit one that is unobserved (unless we observe the whole sequence). The sampling probability of interest is then given by the representation theorem of de Finetti.

Representation theorem (adaptation of de Finetti): If $\mathbf{X}$ is an exchangeable sequence of binary values (and with $\theta$ defined as above), it follows that the elements of $\mathbf{X}|\theta$ are independent with sampling distribution $X_i|\theta \sim \text{IID Bern}(\theta)$ so that for all $k \in \mathbb{N}$ we have: $$\mathbb{P}(\mathbf{X}_k=\mathbf{x}_k | \theta = c) = \prod_{i=1}^k c^{x_i} (1-c)^{1-x_i}.$$ This particular version of the theorem is adapted from O'Neill (2009), which is itself a minor re-framing of de Finetti's famous representation theorem.

Now, within this IID model, the specific probability $\mathbb{P}(X_i=1|\theta=c) = c$ is just the sampling probability of a positive outcome for the value $X_i$. This represents the probability of a single positive indicator conditional on the Banach limit of the sequence of indicator random variables being equal to $c$.

Since this is an area of interest to you, I strongly recommend you read O'Neill (2009) to see the broader approach used here and how it is contrasted with the frequentist approach. That paper asks some similar questions to what you are asking here, so I think it might assist you in understanding how these things can be framed in an operational manner within the Bayesian paradigm.

How do we justify blending two interpretations of probability in Bayes theorem as if they are equivalent?

I presume here that you are referring to the fact that there are certain limiting correspondences analogous to the "frequentist interpretation" of probability at play in this situation. Bayesians generally take an epistemic interpretation of the meaning of probability (what Bernardo and Smith call the "subjective interpretation"). Consequently, all probability statements are interpreted as beliefs about uncertainty on the part of the analyst. Nevertheless, Bayesians also accept that the law-of-large-numbers (LLN) is valid and applies to their models under appropriate conditions, so it may be the case that the epistemic probability of an event is equivalent to the limiting frequency of a sequence.

In the present case, the definition of the parameter $\theta$ is the Banach limit of the sequence of observable values, so it necessarily corresponds to a limiting frequency. Probability statements about $\theta$ are therefore also probability statements about a limiting frequency for the observable sequence of values. There is no contradiction in this.

Answer 2

IMO you can find equally hard-to-answer philosophical questions like this about the foundations of any branch of maths, not just Bayesian statistics: what does it really mean, deep down, to say that 1+1=2? Not sure I could answer that satisfactorily, but I still confidently use arithmetic.

But my interpretation is: we don't need to think of $\theta$ as a long-run probability. It's the number with the property that if my utility function is linear, then I should be indifferent to the opportunity to pay $\theta$ for a contract which pays out $1$ if the next flip of the coin is heads and 0 if it is tails.

Answer 3

More generally, how do Bayesians interpret P(X=x|θ=c) or P(X≤x|θ=c) for any probability model and does this interpretation pose any challenges when interpreting P(θ=s|x) or P(θ≤s|x)?

$P(X=x\vert \theta=c)$ is the degree of belief ascribed to the outcome $X=x$ conditioned on the fact that $\theta =c$ under the model represented by $P$.

$P(\theta =s \vert X=x)$ is the degree of belief ascribed to the outcome $\theta=s$ conditioned on the fact that $X=x$ under the model represented by $P$.

$P(\Theta = \theta)$ is the degree of belief ascribed to the outcome $\Theta=\theta$ under the model represented by $P$.

$P(\Theta)$ is the degree of belief ascribed to the outcome $\Theta=\theta$, for each $\theta \in \Theta$, under the model represented by $P$

(and so on...)

"Degree of belief" can be operationalized in terms of betting/preference behavior (as mentioned in other answers). More abstractly, Bayesian probability theory is a formalization of aspects of how people reason under uncertainty (at least some of the time...), so it is, itself, a model for belief.

How do we justify blending two interpretations of probability in Bayes theorem as if they are equivalent? How does Bayes theorem not imply there is only one type of probability? How are we able to apply posterior probability statements to the unknown fixed true θ=c under investigation?

I believe the first two questions miss a key point: more than one category of phenomena can be modeled using the same mathematics. In this case both (an idealization of) belief and (an idealization of) repeatable trials can both be represented by the same mathematical formulation: probability theory. Good/useful/accurate beliefs about repeatable trials will assign "degrees of belief" for an outcome $x$ that are equal to the proportion of occurrences for the outcome $x$ in the ensemble of trials (basically by Dutch Book arguments), so under those circumstances you get a numerical correspondence between these two different aspects of the world.

For the third question, these statements are "the degree of belief ascribed to the outcome...".

Answer 4

I'm going to go in an entirely different direction from the other replies; I hope this comes across as helpful rather than simply contrary.

I suggest that we don't need to interpret conditional probability (or any other probability) in general. In any actual application, the interpretation of the conditional probability will be forced upon us by the context (in fact, we may be presented with many ways to frame the problem, all with different meanings!).

But without a specific application, the question of interpretation is meaningless. The reason your example seems difficult to interpret is because it is critically underspecified - the problem does not give us any real-world scenario that we're trying to reason uncertainly about. It's not surprising, in such a situation, that it's hard to resolve what exactly we mean by uncertain knowledge.

Answer 5

I think that the issue here is whether the likelihood that is used to turn a Bayesian prior into a Bayesian posterior has a Bayesian interpretation, or whether it is necessarily a long-run frequency. IF that is the question, then I would say "yes", of course it does.

If we have a Bernoulli likelihood then $f(k;\theta) = \theta^k(1-\theta)^{1-k}$ for $k \in \{0,1\}$, where $\theta$ is a parameter, with unknown "true" probability $c$. Now I would regard $c$ as a "physical probability" (IIRC Good uses that term as well). If we are talking about macroscopic objects, like coins, then there is no such thing as "randomness". Whether a coin comes down heads of tails is entirely deteministic. The only reason we can't know whether it comes down heads or tails is that we don't have perfect knowledge of the initial condtions. So a "physical probability" is basically summarising appearance of randomness that are caused by that lack of knowledge.

It is important to distinguish physical probabilities from either Bayesian probabilities or frequentist ones. It distinct from frequentist probabilities because the long run frequency normally stems directly from the physical probability being (assumed to be) the same for all possible trials. However, they are not directly equivalent. The physical probability is directly the probability that a particular coin flip will come up heads, because it describes the physics that makes that so. A frequentist probability, as a long-run frequency can't be applied to a particular coin flip (as it has no long run frequency, it happens only once). It can only make a probabilistic statement about a (fictitious) population of coin flips, of which this can be considered a particular sample.

Note that something that can only ever happen at most once can have a physical probability of happening, but it can't have a long run frequency.

A physical probability isn't a Bayesian probability either. Bayesian probabilities represent our beliefs (subjective or objective) about the plausibility of different values of that physical probability. The Bayesian is making a probabilistic model of the physics, it isn't the physics itself.

So for me, I would view $f(k;c)$ as neither Bayesian nor frequentist, or perhaps both, but a statement about the physics of the date generating process. It is equally true for single observations, or when looking at a long run of observations.

So in this case, to get our posterior, we would say

$$p(\theta|X=1) = \frac{P(x=1;\theta)p(\theta)}{P(X=1)}$$

Note that $c$ does not appear anywhere in our inference to obtain our posterior belief, which is why I think the question as posed is essentially meaningless.

Of course we could look at our posterior to evaluate $p(c|X=1)$, but this would just give us the point probability of the model parameter, $\theta$, being the same numeric value as the true "physical" probability. As this is evaluating a specific point on a continuous distribution, it is just a density, not an actual probability. Probably not very useful.

At the end of the day, it is a very badly posed question, but that seems to be the most meaningful answer I can give to the most informative intepretation that is reasonably consistent with the question as posed.

Answer 6

You play a coin flip game with your friend, but you know that your friend somehow tends to toss heads almost every time. So, you can say something like "Ha I know my friend (prior belief), he always tosses heads, so the probability of him toss heads again will be somewhere between $[0.7,0.9]$", i.e. $\theta$ can be whatever value you want inside that interval.

A natural way for saying that in a probabilistic language, that the probability of success (for your friend) lies inside $[0.7,0.9]$ is to let $\theta$ be a random variable.

Now that $\theta$ is a random variable you can assign it a distribution. But your distribution has to reflect your prior belief, that the probability of success for your friend will be inside the interval $[0.7,0.9]$.

A good choice for distribution for $\theta$ would be a $Beta(a,b)$ distribution (as it takes values inside $[0,1]$ where probabilities also do).

However, this $Beta(a,b)$ distribution must give more attention to values inside $[0.7-0.9]$ which is your prior belief right, that your friend always toss heads.

To do that you can center the distribution around $0.8$ which is the midpoint of the interval $[0.7-0.9]$

You can do that by solving $\frac{a}{a+b}=0.8$ a potential solution for that can be choosing $a=10$ and then $b=2.5$.

So, $\pi(\theta)= Beta(\theta;10,2.5)$ reflect your prior belief about the success probability of your friend that lies inside the interval $[0.7-0.9]$.

Now if you want to say something about as $n$ (the number of samples tends to infinity) then check that the mean of the posterior is

$$Mean = \frac{a+\sum x}{a + \sum x + b +n - \sum x} = \frac{a}{b+n} + \frac{\sum x}{b+n}$$

where for $n\rightarrow \infty$, the mean of your posterior belief $\pi(\theta|x)$ goes to $\bar{x}$

Answer 7

I think part of the problem is that there are some notational problems in the question, and a degree of people talking past each other due to having different backgrounds/positions, so I'll go through the question trying to understand what was meant. I will be happy to be corrected if I am wrong and will edit the answer until we understand each other.

The first issue is what does the author mean by $P(X=x|\theta=c)$? I think this is intended to mean the probability that the random variable $X$ has the value $x$ if the parameter of the model, $\theta$ has its "true" value, $c$.

How do Bayesians interpret θ=c, the probability of heads? θ of course is an unrealized or unobservable realization of a random variable,

This is a problematic line for me as $\theta$ is not a random variable, but a parameter of the model. If we knew what $c$ was, we would just set $\theta = c$ and there would be no need for a prior or a posterior. But we don't know the optimal value of the parameter, so what do we do?

The traditional Bayesian approach is to contsruct a prior for the unknown parameter value, $\pi(\theta)$ that represents what we know about the parameter a-priori (which may be very little). If we want to know what values of $\theta$ are plausible, given our prior and our data point, $X = x$, then we use Bayes rule, giving

$p(\theta|X = x) = \frac{P(X = x|\theta)p(\theta)}{P(X=x)}$

Notice I have written $P(X=x|\theta)$ rather than $P(X = x|\theta = c)$. This is because we are not interested in a single number telling us the probability of a head. We want to continue representing our knowledge in the form of a distribution of relative plausibilities of all possible values of $\theta$. Representing knowledge in the form distributions, rather than point values is fairly central to Bayesianism.

IF we wanted to give a single number representing the probability of a head, then we might take the mode of $P(\theta|X=x)$ or the expectation of $\theta$ with respect to $P(\theta|X=x)$. But asking how Bayesians interpret $\theta = c$ seems meaningless, it is just setting a parameter of our model to a particular value.

For instance, if P(X=1|θ=c)=c is my belief that the coin will land heads, then π(θ) is my belief about my belief, and in some sense so too is the prior predictive distribution P(X=1)=∫θπ(θ)dθ=aa+b. To say "if θ=c is known" is to say that I know my own beliefs. To say "if θ is unknown" is to say I only have a belief about my beliefs.

This seems very confused. In the case of flipping a coin (a Bernoulli trial), then $P(X=1|\theta=c) = c$ is a tautology as the parameter of a Bernoulli distribution is the probability that $X=1$, so this equation only holds when the parameter of the distribution is equal to its true value. But we don't know the value of $c$, so Bayesians wouldn't encounter this. $\theta$ is a parameter of a model, $c$ is it's true value, what more could there be?

$P(X=1|θ=c)=c$ is not my belief that the coin will land heads, it is the true probability that it will land heads. It can't be my belief as it relies on me knowing the correct value of the parameter $\theta$, but I don't. This means that "then π(θ) is my belief about my belief," is incorrect, because the premise was incorrect. It is just your belief about the relative plausibilities of different values of the parameter $\theta$.

To say "if θ=c is known" is to say that I know my own beliefs.

No, this would be equivalent to saying that you know the true value of the parameter $\theta$, so it is just saying the prior should be a delta function centered on $c$. It is just a direct statement of your prior belief/state of knowledge.

To say "if θ is unknown" is to say I only have a belief about my beliefs.

Again, this is incorrect because the premise at the start of the paragraph was false. It just means you don't know the true value of parameter $\theta$ so perhaps a flat prior distribution on the interval 0 to 1 would be appropriate (encoded as a Beta distribution for convenience).

I think I'll leave it at that for now, adding more is likely to just be further talking past eachother, so I will wait for @GeoffreyJohnson 's comments/corrections.

Answer 8

Let us think about what you are attempting to ask. If we define $X=x\in\chi, \chi$ being the sample space, as observed, then in Bayesian thought, it is a constant. It is an observable. Instead of using the language of parameters and data, we can think in terms of observables and unobservables. There is no randomness here.

$\theta$ is normally a random variable in the parameter space, but it is now a constant. It is crucial that we know how it became so. It appears from the language of your posting that we are conditioning on it in the likelihood function so that, for our purposes, the likelihood is now $.82^1$. So it is not a random variable either. So it is senseless to talk about a probability when everything is a constant. It would be like discussing the probability that $2+2=4$.

It isn’t impossible to discuss this, but it is difficult for several reasons. First, the interpretation can change depending on the axioms used to derive Bayes rule. For example, if we are conditioning on $\theta=.82$, then de Finetti’s axioms would require a prior with mass only on .82. What if we were using some other axiomatization and the mass of the prior was zero at the point we conditioned it? Cox’s axioms would find that problematic as well. Savage’s might not if we allowed for time inconsistency, though why you would change your mind on the prior and not the likelihood is beyond me.

We also need a better definition of what a constant is. For example, conditioning some parameters on constants is not that unusual in Bayesian thinking. Sometimes you do know one of them. There is another case, though, that wrenches up even the Frequentist toolset.

To give an example, the speed of light is known precisely. However, as distance is now normed against the speed of light, distance is uncertain. We used to measure the speed of light with uncertainty; we now measure distance with uncertainty.

Let us imagine we get out our carefully built scientific equipment and decide to measure out five kilometers for our morning run. Our equipment is accurate to within plus or minus twenty meters. When our device measures five kilometers, we know it is somewhere within 4980 and 5020 meters in reality. It is close enough. If this is part of our measuring, we could condition on it being five kilometers as it is close enough for our purposes. It is also definitely wrong. Because distance is a value in the real numbers, the probability that our actual distance is five kilometers when it registers five kilometers is a measure zero event. Our conditioning is wrong with certainty.

A second issue with this type of conditioning problem is a non-mathematical issue. If, instead, we were running a wrecking ball and hit our intended building, plus or minus twenty meters, we could be hitting the wrong building. At the same time, we have conditioned our uncertainty away. Had our wrecking ball been run by a robot, a la E.T. Jaynes, we would have no way to know our decision process was bad.

On the surface, you may think that would not matter, but de Finetti’s coherence wrecks that idea if we are gambling money. A bad constant could create a Dutch Book.

As I see it, there is no randomness in your problem. We observed the outcome; it is a certainty. We observed the parameter. It is being treated as a certainty. We are being bigoted in our conditioning in that we are saying there is no uncertainty.

What is probability in the face of perfect certainty? What do you mean by your question?

Answer 9

$P(x=1|\theta=c)=c=1-P(x=0|\theta=c)=\theta_i=P(x=1|\theta_i)$

The $\theta$ is the aleatoric uncertainty inherited from the coin, and before we observe any events we can only guess that a certain fraction would be most possible, for instance 0.5, but $\theta_i$ can take very value between 0 and 1. We assume that it is discrete (for simplification) and can only take values from this list containing 11 values: $\theta_i \in$ [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0].

---

$P(\theta)$

It is a beta distribution, the probability of that probability is high or low given the observations or our belief. It acts as a prior, meaning that $P(\theta)=P(\theta|X)$ for the new observations after we have seen $X$.

Since we assume $\theta$ is discrete, then every $\theta_i$ takes a value/probability representing the probability of $P(x=1|\theta_i)$.

However in frequentist, it is a delta distribution. Say we see three heads in three tossing, then $P(\theta=1)=1$ and 0 elsewhere for $\theta$. Its aleatoric uncertainty is 0. If we observe 3 more events but all with tails, the delta distribution would change with $P(\theta=0.5)=1$ and 0 elsewhere. And $P(\theta=1)$ would also become 0. Its aleatoric uncertainty is 0.5 now.

---

$P(x=1)=\sum_{i=1}^NP(x=1|\theta_i)P(\theta_i)$

Say we don't know the $P(\theta)$, meaning we don't know every $P(\theta_i)$ or the probability of every probability is 1/11. Then

$\begin{align*} P(x=1)&=\sum_{i=1}^{11}P(x=1|\theta_i)P(\theta_i)=\sum_{i=1}^{11}\theta_iP(\theta_i)\\&=\frac{1}{11} \sum_{i=1}^{11}\theta_i=\frac{1}{11} \sum_{c=0}^{1} P(\theta=c)\\&=\frac{1}{11}\sum [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0] =0.5 \end{align*}$

---

$P(\theta|X)$

Following the above and we observe some data $X$. It is a distribution and for every $\theta_i$,

$\begin{align*} P(\theta_i|X)&=\frac{P(X|\theta_i)P(\theta_i)}{\sum_{i=1}^{11}P(X|\theta_i)P(\theta_i)}\\&=\frac{P(x=1|\theta_i)^k P(x=0|\theta_i)^{N-k}P(\theta_i)}{\sum_{i=1}^{11}P(x=1|\theta_i)^k P(x=0|\theta_i)^{N-k}P(\theta_i)}\\&= \frac{P(x=1|\theta_i)^k (1-P(x=1|\theta_i))^{N-k}P(\theta_i)}{\sum_{i=1}^{11}P(x=1|\theta_i)^k (1-P(x=1|\theta_i))^{N-k}P(\theta_i)} \end{align*}$

where k is the number of observations of heads and N is the total tosses.

After we obtained $P(\theta_i|X)$, say we observed some other data $Z$, then to calculate $P(\theta_i|Z)$ we would treat $P(\theta) = P(\theta|X)$ for every 11 values of $\theta_i$. With enough observations from the initial prior $P(\theta_i) = 1/11$ would vanish.

For $P(\theta\leq s|X)$ you just sum up all the $P(\theta_i|X)$ where $\theta_i$ less than or equal to s.

Answer 10

• It's a measure of $X$ be $x$ given $\theta=c$ assuming that I will look through the distribution/estochastic-variability of $\theta$ (notice that depends from my distribution belief) in this particular event $\theta=c$ that is I will measure the odd of $X=x$ assuming both $\theta=c$ and its probabilistic uncertainty, furthermore, the math will show that $P(X=x|\theta=c)$ will depend from my belief to $\theta=c$ because the probability-view of the event $\theta=c$ (that will come up by the math of $P(X=x|\theta=c)$ ) will be my prior distribution of $\theta$ evaluating other belief that looks at $P(\theta=c)$ so $P(X=x|\theta=c)$ will depend from this other belief. I mean you are measuring the odd of $X=x$ taking your belief about $\theta=c$ although this was not implicitly clear.

Answer 11

Below are four different interpretations using the coin toss example that was provided in the original question. Option 1 a) appears to be the appropriate interpretation under the Bayesian paradigm. If you find one of these that maps to your answer, please identify it and offer suggestions for improvement if needed.

Option 1: Probability statements about $X$ and probability statements about $\theta$ are both statements of personal belief.

a) $\theta$ is the limiting proportion of heads as the number of flips tends to infinity for the coin under investigation, an unknown fixed constant with value $c$ and is not a probability. The only valid probability is my belief about any given flip, which I set equal to this unknown fixed constant. Therefore, $P\{X=1|\theta=c\}=c$ is my personal belief about the coin landing heads in any given throw if I know this limiting proportion. Since I do not know what to believe, $\pi(\theta)$ is my belief about the limiting proportion (not my belief about the probability of heads). If I were to integrate the data pmf using the prior distribution I would get the prior predictive distribution. Then $P\{X=1\}=\frac{a}{a+b}$ where $\frac{a}{a+b}$ is a "known" constant. In a different sense this would be my belief about the coin landing heads when not knowing the limiting proportion. The posterior is my belief about the limiting proportion given the observed data. Nevertheless, the prior and posterior probabilities do not represent factual statements about the limiting proportion of heads for the coin under investigation, nor are they statements about the experiment. Option 1 a) amounts to Option 2 b) since this is how the posterior is operationalized in practice using Monte Carlo simulations.

b) $c$ is the limiting proportion of heads as the number of flips tends to infinity for the coin under investigation, an unknown fixed constant and not a probability. The only valid probability is my belief about any given flip, which I set equal to this unknown fixed constant. Therefore $\theta=c$, and equivalently $P\{X=1|\theta=c\}=c$, is my personal belief about the coin landing heads in any given throw if I know this limiting proportion. Since I do not know what to believe, $\pi(\theta)$ is my belief… about my belief. If I were to integrate the data pmf using the prior distribution I would get the prior predictive distribution. Then $P\{X=1\}=\frac{a}{a+b}$ where $\frac{a}{a+b}$ is a "known" constant. In a different sense this would also be my belief about my belief. This option has us applying a belief probability measure to a belief probability measure. Similarly for $\pi(\theta|\boldsymbol{x})$ and $P\{X=1|\boldsymbol{x}\}$. However, because of my original correspondence we can interpret prior and posterior probabilities as beliefs about the limiting proportion of heads for the coin under investigation. Nevertheless, these prior and posterior probabilities do not represent factual statements about the limiting proportion of heads for the coin under investigation, nor are they statements about the experiment. Option 1 b) amounts to Option 2 b) since this is how the posterior is operationalized in practice using Monte Carlo simulations.

Option 2: Probability statements about $X$ and probability statements about $\theta$ both have a frequentist interpretation.

a) $\theta=c$, and equivalently $P\{X=1|\theta=c\}=c$, is the limiting proportion of heads as the number of flips tends to infinity for the coin under investigation, an unknown fixed constant. The density $\pi(\theta)$ depicts a collection of $\theta$’s (coins) or the limiting proportions of randomly selected $\theta$’s (coins) as the number of draws from $\pi(\theta)$ tends to infinity. These probabilities are considered known constants. The unknown true $\theta=c$ under investigation was randomly selected from the known collection or prevalence of $\theta$’s according to $\pi(\theta)$, and the observed data is used to subset this collection forming the posterior. If we are to apply these posterior probability statements to make inference on the unknown true $\theta$ (coin) under investigation we have to change our sampling frame. We must imagine instead that the unknown true $\theta$ was instead randomly selected from the posterior. This, then, has cause and effect reversed since the posterior distribution, from which we selected $\theta$, depends on the data… but the data depended on the $\theta$ we had not yet selected from the posterior. We could imagine drawing a new $\theta$ (coin) from the posterior, but this would not be the same $\theta=c$ we started with under investigation. The challenge here is applying the probability statement in the posterior distribution to the unknown true $\theta$ (coin) under investigation in a meaningful way.

b) $\theta=c$, and equivalently $P\{X=1|\theta=c\}=c$, is the limiting proportion of heads as the number of flips tends to infinity for the coin under investigation, an unknown fixed constant. The density $\pi(\theta)$ depicts a collection of other $\theta$’s (coins) I have given to myself or the limiting proportions of randomly selected $\theta$’s (coins) as the number of draws from $\pi(\theta)$ tends to infinity. These probabilities are considered known constants. The observed data is used to subset this collection forming the posterior. The posterior is a legitimate sampling distribution of $\theta$'s (coins) I have given myself. The challenge here is applying the probability statements in the posterior distribution to the unknown true $\theta$ (coin) under investigation in a meaningful way since at no point was the true $\theta$ (coin) sampled from the posterior.

Option 3: Probability statements about $X$ have a frequentist interpretation and probability statements about $\theta$ represent personal belief.

$\theta=c$, and equivalently $P\{X=1|\theta=c\}=c$, is the limiting proportion of heads as the number of flips tends to infinity for the coin under investigation. Since I do not know this limiting proportion, $\pi(\theta)$ is my personal belief about this unknown fixed quantity. On the surface this seems the most reasonable. However, this would have Bayes theorem blending two different interpretations of probability as if they are compatible or equivalent, and it does not provide a clear link between posterior probability and the unknown fixed true $\theta$ under investigation. This would mean we are dealing with Option 1 or Option 2. Even if one insists on two different yet compatible interpretations of probability, Option 3 amounts to Option 2 b) since this is how the posterior is operationalized in practice using Monte Carlo simulations.

Option 4: Probability statements about $X$ have a frequentist interpretation and there are no probability statements about $\theta$.

$\theta=c$, and equivalently $P\{X=1|\theta=c\}=c$, is the limiting proportion of heads for the coin under investigation. The reason Bayesian statistics can provide reasonable point and interval estimates despite the shortcomings above regarding interpretation is that at the core of every prior is a likelihood. Something was witnessed or observed that gave rise to a likelihood, and therefore the prior. There are in fact no probability statements about $\theta$, belief, long-run, or otherwise. Bayes theorem amounts to multiplying independent likelihoods equivalent to a fixed effect meta-analysis, except the Bayesian normalizes the joint likelihood instead of inverting a hypothesis test. If we view the Bayesian inference machine as a frequentist meta-analytic testing procedure the shortcomings above vanish. The posterior is an asymptotic confidence distribution. Bayesian belief is more objectively viewed as confidence based on frequency probability of the experiment.