Intuition of terms P(θ) and P(y) in Bayes' Theorem [duplicate]

Question

I am struggling with the intuition behind terms in the Bayes' Theorem. In the simple example of a deck of cards we have:

$$ P(King | Red) = \frac{P(Red | King)P(King)}{P(Red)} $$

The terms in this equation are intuitive to me. P(Red | King) is the probability a card is red given that it is a king. P(King) and P(Red) is just the probability of a card being a king or red (1/13 or 1/2 respectively).

I am trying to apply this intuition to a more complex example, consider some data that we assume is distributed binomially with probability parameter theta, we observe 10 trials (N=10). This gives: $$ y \sim Binomial(N,\theta) $$ Which implies that: $$ p(y | \theta) = \binom{N}{y} \theta^y (1 - \theta)^{N-y} $$ And again we get Bayes' Formula:

$$ P(\theta | y) = \frac{P(y | \theta)P(\theta)}{P(y)} $$

The P(y | theta) term can be thought of as a surface (see pic below).

However, what is the intuition behind the remaining two terms $P(\theta)$ and $P(y)$?

What is the probability that $\theta = 0.4$? The space is no longer discrete like the simple example and so this doesn't really have meaning. This leads me to believe that these terms now relate to integrals over the surface in the diagram above. Is my understanding correct?

Answer 1

$P(\theta)$ and $P(y)$ are priors; you can easily apply the same intuition you have for $P(y|\theta)$ by pretending they are conditionals $P(y|Z)$ and $P(\theta|Z)$ where Z is a latent variable you don't care about - if you describe $P(y|\theta)$ is a surface, they, too are surfaces.

The 3D cube you've shown assumes that $y$ and $\theta$ are flat surfaces, and they can be - but since they can be functions of another variable, they too can be warped.

Geometrically, this would push the joint density curves up or down, depending on whether the warping produces a 'hill' or a 'sinkhole', and since the elevation of the curve represents probability, make the corresponding interval of values accordingly more or less likely.

Conditioning on the variable values corresponds to taking a slice of the cube cut along the axis representing that variable, e.g. $p(y,\theta|y=4)$ picks and isolates one of the colored curve graphs - in this case, the one whose endpoint crosses $y$ at 4 (I believe it's the first red curve).

Now, something like the probability $p(\theta=0.4)$ is indeed a bit arbitrary. The prior probability of the parameters depends... more or less on how much reason you have to favor some values over others.

This is both a weakness and a strength. A bad prior may warp your results, and it's not something trivially verifiable. You can use a uniform prior to represent total ignorance, but sometimes a prior too far from the actual value is a problem. If you make a strong assumption and you're wrong, you will throw out perfectly correct conclusions as unlikely. Your reasoning is only as strong as your model.

On the flip side, it represents the modularity of the Bayesian updating. Your priors in one experiment may be your posteriors from a previous one, and if your high-level assumptions are correct (e.g. you don't suddenly get negative counts for something you thought was Binomial), the updates will asymptotically update to the true value.

Answer 2

On their own, they are the probabilities of $\theta $ and $y$. But why are they there?

The logic all comes from the definition of conditional probabilities:

$$ P(y \vert \theta) = P(y,\theta)/P(\theta)$$

Hence, multiplying this by $P(\theta)$ returns us the joint probability. I like to think of it like this: $P(y \vert \theta)$ is the probability of $y$ given $\theta$ as realized. So, if you tell me $\theta = 3$ I can now compute the probability of $y$ conditioned on it. But how often will this happen? How often will I be in a situation where $\theta = 3$? And consequently, what's the probability of getting a certain $y$ when $\theta = 3$ ?

Why we divide by $P(y)$? Well, that is because we want to condition on it. In general

$$ P(A \vert B) = P(A,B)/P(B) $$

The division acts as to scale the probability space of both $A$ and $B$ to the probability where $B$ has happened.

Remember that Bayes theorem is just a mechanical trick to rewrite a specific conditional probability (that we might not know) to some other probabilities (that hopefully we can compute or estimate).

You are right to wonder about the meaning of a probability sign in continuous versus discrete setting, but don't be concerned. You can compute probabilities of continuous rv with integral, which makes them a little more complicated, but everything will keep working with relatively simple adjustments. You might want to do some simple exercises on computing probabilities for cont. rv and in particular on conditional probabilities with CRV.

Answer 3

To understand the intuition behind the prior and the denominator, it may help to solve a problem with them.

Let’s first start with the prior. The priors you have been using are all nice priors that are not really “real.” While it is okay to construct a conjugate prior distribution which is close to your beliefs but it is not okay to use a prior because it is convenient or you have not thought about the problem.

So let us imagine that you love to play Cho Han, and a recent law governing casinos has been passed, and there are very many casinos in your area. They are perfectly competitive and the game is supposed to be played so that the long-run probability of winning is fifty percent.

For it to be worth it to the casino to offer the game, they must create a cover charge as that is their profit. If you want an instructive lesson on our problem, watch the 1962 Japanese movie Zatoichi about a blind swordsman that enters a mob casino at the beginning of the film.

The problem with fair gambles such as a coin toss is that there is no such thing. If you flip a coin and do not know the outcome ahead of time, then you would make a terrible magician, con man, or physicist. Fair coins can generate an infinite number of heads or an endless number of tails in a row if you know what you are doing.

Now let us imagine there are two types of casinos. The first type is an honest casino made up of croupiers that have passed careful background checks. The second type is a casino that only hires magicians and con men. They have to control the rate of winning so that they remain beneath the regulator’s radar and stay open.

Both types of casinos also monitor their croupiers to detect any that may be cheating in the players’ favor to get tips or out of spite to hurt their employer.

The prior from zero to one is not a mass function; it is a density. A density is not a probability because at any point there is no width, so the probability is zero. If the density one point is 4 and another 2, then there the relative odds of the first point is twice that of the second. We need to convert beliefs into a function.

It may help to think of the parameter as part of a parameter space, $\theta\in\Theta$. The goal is to construct relative positions across the space.

Let us start with the easy part of the problem, the case where $\theta<.5$ so that the casino is losing systematically. The croupier could make small losses to get tips or significant losses to get even at a manager or owner. Nonetheless, it should be a rare outcome. Although one percent is too high, to make the example easy, we are going to set the probability at one percent. Since we have no sense of location, it will be one percent uniformly distributed over the half-open interval $[0,.5)$.

On the other side, we want to ask what range of $\theta$ is very improbable. At $\theta=.55$, it is going to become apparent over a large enough group. Also, some casinos are going to be fair at $.5.$ The value is likely closer to $.5$ than $.53$ Some function that is decreasing as it goes to the right and is assured to be a density is needed. As we railed against above, we are going to choose the normal density truncated on the left at $.5$ and the right at $1.$ where $\sigma=.01$ so that most of the mass is in the appropriate range.

The resulting prior is $$\Pr(\theta)=\begin{cases}1/50&\theta<.5\\ \mathcal{N}(.5,.01)\frac{99}{50}&\text{ otherwise.}\end{cases}$$

We may quibble a little here or there on the function but once it has been decided, it is fixed. Can it change? Sure, if there is new information from outside the sample. Otherwise, the prior is the prior. The intuition around $\Pr(y)$ is that it is a constant. Because $\theta$ is marginalized out, it does not depend on the actual value of $\theta$. It is unique to the sample because it is the probability of seeing the sample.

Given the information in the prior, we can look at how $\Pr(y)$ changes with the number of successes and failures in a sample.

This does not mean that as successes increase that $\Pr(y)$ increases, but it does over this narrow range. Instead, if the graph of the numerators is viewed, with the sample of 47 successes as the solid line and 53 successes and the heavy line you can see that there is simply more area.

This is deceptive because the posterior doesn't look so close as shown in the posteriors below.

One last element of intuition regarding the prior is the subjectivity. The weakness and the strength of the prior is the validity of the information. Imagine that, unknown to the gambler, the owner of the casino died or was hospitalized and the croupier wanted to drive the family bankrupt in a non-obvious way. The player wins 5200 times and loses 4800.

I didn't calculate the posterior because it would require a bit more work because of the scaling, but the plot of the numerator's log density below shows that the prior introduces a lot of prejudice.

The prior is still impacting the result even after 10,000 observations. The maximum a posteriori estimator is $\theta=.5$ and it is not $\theta=.48$. Had this not been done in log density, then there would have been two very narrow spikes. Those spikes would be very tall and $\Pr(y)$ would be vanishingly small.

That is why prejudice is such a powerful force, it can overwhelm data.

Answer 4

However what is the intuition behind the remaining two terms $P(\theta)$ and $P(y)$ ?

I like to view Bayes formula as getting a marginal conditional distribution out of a joint distribution.

$$P(\theta \vert y) P(y)= P(y,\theta) = P(y \vert \theta) P(\theta)$$

where $P(y)$ and $P(\theta)$ are marginal distributions.

Example:

Say your card deck is a bit weird (otherwise the example is not interesting) and has 60k cards distributed as:

          hearts   diamonds   clubs   spades
king      1/60     1/60       1/15    1/15          1/6
queen     1/30     1/30       1/60    1/60          1/10
jack      1/60     1/60       1/30    1/30          1/10
10        1/15     1/15       1/30    1/30          1/5 
9         1/6      1/60       1/60    1/60          1/15 
8         1/20     1/20       1/30    1/30          1/6
7         1/20     1/10       0/60    1/20          1/5

          1/4      3/10       1/5     1/4

The numbers are:

• joint distributions in the matrix: For instance, the number 1/60 in the upper left corner is the probability to draw a king of hearts.
• marginal distributions on the right side and bottom side: For instance, the number 1/6 on the upper right is the probability to draw a king, and the number 1/4 on the bottom left is the probability to draw a hearts.

The conditional distribution is found by comparing all the possibilities given the condition. For instance: There are many less red kings (king of hearts and king of diamonds) than black kings (king of clubs and king of spades) in the deck. So, if you have a king, then it is four times more likely that it is a black card (clubs or spades) than a red card (hearts and diamonds). You can compute this as:

$$P(\text{red|king}) = \frac{P(\text{red and king})}{P(\text{king})} = \frac{\overbrace{1/60+1/60}^{\text{probability red and king}}}{\underbrace{1/60+1/60+1/15+1/15}_{\text{total probability (red or any other colour) king}}}$$

So you can write Bayes theorem also as:

$$\begin{array}{} P(\hat\theta \vert y) &=& \frac{P(y \text{ and } \hat\theta)}{\sum_{\forall \theta_i \in \Theta} P(y \text{ and } \theta_i)} \\ &=& \frac{P(y \text{ and } \hat\theta)}{ P(y \text{ and } \theta_1) + P(y \text{ and } \theta_2) + ... + P(y \text{ and } \theta_{n-1}) + P(y \text{ and } \theta_{n})}\\ &=&\frac{P(y \text{ and } \hat\theta)}{P(y)} \end{array} $$

that is, you comparing a single cell $P(y \text{ and } \hat\theta)$ with a sum of $P(y \text{ and } \theta_i)$ for all possible $\theta_i$ (the sum in a row, which equals the marginal probability $P(y)$).

In addition, you could write

$$P(\hat\theta \vert y) = \frac{P(y \text{ and } \hat\theta)}{P(y)} = \frac{P(y \vert \hat\theta)P(\theta)}{P(y)}$$

you could see this as

• $P(y \vert \hat\theta)$ is the likelihood. It gives how likely the result/observation $y$ is given the theory $\theta$. The higher this number relatively is, the more the particular observation is support for the particular theory.
• $P(\theta)$ this is the prior probability of $\theta$ in some cases this is not really known and people call it prior believes. The likelihood should not be considered in isolation. Some high likelihood is "proof" for a given theory, but one must also consider it in conjunction with the probaility of the theory. In your question, you write that $P(y \vert \theta)$ can be thought of a surface, but that is not a surface depicting joint probability. You would need to scale the different lines, $P(y \vert \theta_i)$, in the surface for each $\theta_i$ according to the probability $P(\theta_i)$
• $P(y)$ you could see this as a normalization constant. For example: if you look in the table te first row, the 1/60, 1/60, 1/15, 1/15, are joint probabilities for kings of hearts, kings of diamonds, kings of clubs and kings of spades. They give the relative conditional probability for hearts, diamonds, clubs and spades, if the card is a king. But, you can not just say that 1/60 is the probability for hearts given that the card is king. You first need to go from relative conditional probabilities to absolute conditional probabilities by scaling it by the sum 1/60+1/60+1/15+1/15.