2
$\begingroup$

I am struggling with the intuition behind terms in the Bayes' Theorem. In the simple example of a deck of cards we have:

$$ P(King | Red) = \frac{P(Red | King)P(King)}{P(Red)} $$

The terms in this equation are intuitive to me. P(Red | King) is the probability a card is red given that it is a king. P(King) and P(Red) is just the probability of a card being a king or red (1/13 or 1/2 respectively).

I am trying to apply this intuition to a more complex example, consider some data that we assume is distributed binomially with probability parameter theta, we observe 10 trials (N=10). This gives: $$ y \sim Binomial(N,\theta) $$ Which implies that: $$ p(y | \theta) = \binom{N}{y} \theta^y (1 - \theta)^{N-y} $$ And again we get Bayes' Formula:

$$ P(\theta | y) = \frac{P(y | \theta)P(\theta)}{P(y)} $$

The P(y | theta) term can be thought of as a surface (see pic below).

enter image description here

However, what is the intuition behind the remaining two terms $P(\theta)$ and $P(y)$?

What is the probability that $\theta = 0.4$? The space is no longer discrete like the simple example and so this doesn't really have meaning. This leads me to believe that these terms now relate to integrals over the surface in the diagram above. Is my understanding correct?

$\endgroup$
2
  • $\begingroup$ Useful link: rpubs.com/richarddmorey/binomial-beta $\endgroup$
    – JDraper
    Jan 24, 2020 at 16:05
  • 1
    $\begingroup$ I do not find the other duplicate question equivalent. There is a nice answer from Ben, but it takes an entirely different swing/form/interpretation and does not explain the P(θ) and P(y) individually. $\endgroup$ Jan 26, 2020 at 14:15

4 Answers 4

1
$\begingroup$

$P(\theta)$ and $P(y)$ are priors; you can easily apply the same intuition you have for $P(y|\theta)$ by pretending they are conditionals $P(y|Z)$ and $P(\theta|Z)$ where Z is a latent variable you don't care about - if you describe $P(y|\theta)$ is a surface, they, too are surfaces.

The 3D cube you've shown assumes that $y$ and $\theta$ are flat surfaces, and they can be - but since they can be functions of another variable, they too can be warped.

Geometrically, this would push the joint density curves up or down, depending on whether the warping produces a 'hill' or a 'sinkhole', and since the elevation of the curve represents probability, make the corresponding interval of values accordingly more or less likely.

Conditioning on the variable values corresponds to taking a slice of the cube cut along the axis representing that variable, e.g. $p(y,\theta|y=4)$ picks and isolates one of the colored curve graphs - in this case, the one whose endpoint crosses $y$ at 4 (I believe it's the first red curve).

Now, something like the probability $p(\theta=0.4)$ is indeed a bit arbitrary. The prior probability of the parameters depends... more or less on how much reason you have to favor some values over others.

This is both a weakness and a strength. A bad prior may warp your results, and it's not something trivially verifiable. You can use a uniform prior to represent total ignorance, but sometimes a prior too far from the actual value is a problem. If you make a strong assumption and you're wrong, you will throw out perfectly correct conclusions as unlikely. Your reasoning is only as strong as your model.

On the flip side, it represents the modularity of the Bayesian updating. Your priors in one experiment may be your posteriors from a previous one, and if your high-level assumptions are correct (e.g. you don't suddenly get negative counts for something you thought was Binomial), the updates will asymptotically update to the true value.

$\endgroup$
5
  • $\begingroup$ Thanks for this explanation. If i have understood correctly, can p(y,θ|y=4) be interpreted as the area contained within that plane? (i.e. the integral of the distribution at y=4 between theta = 0 and theta =1). Which means you can extend this to the p(θ=0.4) case (integrating the plane that lies on a tangent to the previous plane). $\endgroup$
    – JDraper
    Jan 24, 2020 at 15:08
  • $\begingroup$ You also initially talk about some other latent variable Z. Thinking about this in terms of a hierarchical Bayesian model, can this latent variable be considered to be a upper level or hyper parameter? $\endgroup$
    – JDraper
    Jan 24, 2020 at 15:12
  • 1
    $\begingroup$ re: 1, it's a bit awkward to talk about, because we're accounting for all the degrees of freedom. It is an integral, but in this specific case it integrates to 1 everywhere by the law of total probability. If we were talking about $p(y|N,\theta)$, there would be a nice mapping to integrating out one of the variables. $\endgroup$
    – jkm
    Jan 24, 2020 at 15:27
  • 1
    $\begingroup$ The Z is whatever, really. If it's a single value, it's a hyperparameter. If it's an abstract distribution of values, it's a hyperprior. It could be a set of stochastic or deterministic variables with their own ancestors somewhere in the model. $\endgroup$
    – jkm
    Jan 24, 2020 at 15:31
  • $\begingroup$ I think I am beginning to understand, although I am not there yet. I'll take a look at some more reading material and ask another question if needed. Thanks for your help. $\endgroup$
    – JDraper
    Jan 24, 2020 at 16:05
2
$\begingroup$

On their own, they are the probabilities of $\theta $ and $y$. But why are they there?

The logic all comes from the definition of conditional probabilities:

$$ P(y \vert \theta) = P(y,\theta)/P(\theta)$$

Hence, multiplying this by $P(\theta)$ returns us the joint probability. I like to think of it like this: $P(y \vert \theta)$ is the probability of $y$ given $\theta$ as realized. So, if you tell me $\theta = 3$ I can now compute the probability of $y$ conditioned on it. But how often will this happen? How often will I be in a situation where $\theta = 3$? And consequently, what's the probability of getting a certain $y$ when $\theta = 3$ ?

Why we divide by $P(y)$? Well, that is because we want to condition on it. In general

$$ P(A \vert B) = P(A,B)/P(B) $$

The division acts as to scale the probability space of both $A$ and $B$ to the probability where $B$ has happened.

Remember that Bayes theorem is just a mechanical trick to rewrite a specific conditional probability (that we might not know) to some other probabilities (that hopefully we can compute or estimate).

You are right to wonder about the meaning of a probability sign in continuous versus discrete setting, but don't be concerned. You can compute probabilities of continuous rv with integral, which makes them a little more complicated, but everything will keep working with relatively simple adjustments. You might want to do some simple exercises on computing probabilities for cont. rv and in particular on conditional probabilities with CRV.

$\endgroup$
3
  • $\begingroup$ I am uncertain this answer the OP question about the intrinsic meaning of $p(\theta)$, the prior density of the probability $\theta$. $\endgroup$
    – Xi'an
    Jan 24, 2020 at 12:55
  • $\begingroup$ I think there are two questions: one is on interpretation of the two marginal probabilities and one is on how does the probability of a cont. RV works, which probably deserves a different question $\endgroup$
    – Three Diag
    Jan 24, 2020 at 12:57
  • $\begingroup$ Thanks for the answer, I accepted @jkm as i thought it answered my question better (although I accept my questions was poorly defined!) $\endgroup$
    – JDraper
    Jan 24, 2020 at 15:13
1
$\begingroup$

To understand the intuition behind the prior and the denominator, it may help to solve a problem with them.

Let’s first start with the prior. The priors you have been using are all nice priors that are not really “real.” While it is okay to construct a conjugate prior distribution which is close to your beliefs but it is not okay to use a prior because it is convenient or you have not thought about the problem.

So let us imagine that you love to play Cho Han, and a recent law governing casinos has been passed, and there are very many casinos in your area. They are perfectly competitive and the game is supposed to be played so that the long-run probability of winning is fifty percent.

For it to be worth it to the casino to offer the game, they must create a cover charge as that is their profit. If you want an instructive lesson on our problem, watch the 1962 Japanese movie Zatoichi about a blind swordsman that enters a mob casino at the beginning of the film.

The problem with fair gambles such as a coin toss is that there is no such thing. If you flip a coin and do not know the outcome ahead of time, then you would make a terrible magician, con man, or physicist. Fair coins can generate an infinite number of heads or an endless number of tails in a row if you know what you are doing.

Now let us imagine there are two types of casinos. The first type is an honest casino made up of croupiers that have passed careful background checks. The second type is a casino that only hires magicians and con men. They have to control the rate of winning so that they remain beneath the regulator’s radar and stay open.

Both types of casinos also monitor their croupiers to detect any that may be cheating in the players’ favor to get tips or out of spite to hurt their employer.

The prior from zero to one is not a mass function; it is a density. A density is not a probability because at any point there is no width, so the probability is zero. If the density one point is 4 and another 2, then there the relative odds of the first point is twice that of the second. We need to convert beliefs into a function.

It may help to think of the parameter as part of a parameter space, $\theta\in\Theta$. The goal is to construct relative positions across the space.

Let us start with the easy part of the problem, the case where $\theta<.5$ so that the casino is losing systematically. The croupier could make small losses to get tips or significant losses to get even at a manager or owner. Nonetheless, it should be a rare outcome. Although one percent is too high, to make the example easy, we are going to set the probability at one percent. Since we have no sense of location, it will be one percent uniformly distributed over the half-open interval $[0,.5)$.

On the other side, we want to ask what range of $\theta$ is very improbable. At $\theta=.55$, it is going to become apparent over a large enough group. Also, some casinos are going to be fair at $.5.$ The value is likely closer to $.5$ than $.53$ Some function that is decreasing as it goes to the right and is assured to be a density is needed. As we railed against above, we are going to choose the normal density truncated on the left at $.5$ and the right at $1.$ where $\sigma=.01$ so that most of the mass is in the appropriate range.

The resulting prior is $$\Pr(\theta)=\begin{cases}1/50&\theta<.5\\ \mathcal{N}(.5,.01)\frac{99}{50}&\text{ otherwise.}\end{cases}$$

prior

We may quibble a little here or there on the function but once it has been decided, it is fixed. Can it change? Sure, if there is new information from outside the sample. Otherwise, the prior is the prior. The intuition around $\Pr(y)$ is that it is a constant. Because $\theta$ is marginalized out, it does not depend on the actual value of $\theta$. It is unique to the sample because it is the probability of seeing the sample.

Given the information in the prior, we can look at how $\Pr(y)$ changes with the number of successes and failures in a sample.

pofy

This does not mean that as successes increase that $\Pr(y)$ increases, but it does over this narrow range. Instead, if the graph of the numerators is viewed, with the sample of 47 successes as the solid line and 53 successes and the heavy line you can see that there is simply more area.

numerators

This is deceptive because the posterior doesn't look so close as shown in the posteriors below.

posteriors

One last element of intuition regarding the prior is the subjectivity. The weakness and the strength of the prior is the validity of the information. Imagine that, unknown to the gambler, the owner of the casino died or was hospitalized and the croupier wanted to drive the family bankrupt in a non-obvious way. The player wins 5200 times and loses 4800.

I didn't calculate the posterior because it would require a bit more work because of the scaling, but the plot of the numerator's log density below shows that the prior introduces a lot of prejudice.

prejudice from prior

The prior is still impacting the result even after 10,000 observations. The maximum a posteriori estimator is $\theta=.5$ and it is not $\theta=.48$. Had this not been done in log density, then there would have been two very narrow spikes. Those spikes would be very tall and $\Pr(y)$ would be vanishingly small.

That is why prejudice is such a powerful force, it can overwhelm data.

$\endgroup$
1
$\begingroup$

However what is the intuition behind the remaining two terms $P(\theta)$ and $P(y)$ ?

I like to view Bayes formula as getting a marginal conditional distribution out of a joint distribution.

$$P(\theta \vert y) P(y)= P(y,\theta) = P(y \vert \theta) P(\theta)$$

where $P(y)$ and $P(\theta)$ are marginal distributions.

Example:

Say your card deck is a bit weird (otherwise the example is not interesting) and has 60k cards distributed as:

          hearts   diamonds   clubs   spades
king      1/60     1/60       1/15    1/15          1/6
queen     1/30     1/30       1/60    1/60          1/10
jack      1/60     1/60       1/30    1/30          1/10
10        1/15     1/15       1/30    1/30          1/5 
9         1/6      1/60       1/60    1/60          1/15 
8         1/20     1/20       1/30    1/30          1/6
7         1/20     1/10       0/60    1/20          1/5

          1/4      3/10       1/5     1/4

The numbers are:

  • joint distributions in the matrix: For instance, the number 1/60 in the upper left corner is the probability to draw a king of hearts.
  • marginal distributions on the right side and bottom side: For instance, the number 1/6 on the upper right is the probability to draw a king, and the number 1/4 on the bottom left is the probability to draw a hearts.

The conditional distribution is found by comparing all the possibilities given the condition. For instance: There are many less red kings (king of hearts and king of diamonds) than black kings (king of clubs and king of spades) in the deck. So, if you have a king, then it is four times more likely that it is a black card (clubs or spades) than a red card (hearts and diamonds). You can compute this as:

$$P(\text{red|king}) = \frac{P(\text{red and king})}{P(\text{king})} = \frac{\overbrace{1/60+1/60}^{\text{probability red and king}}}{\underbrace{1/60+1/60+1/15+1/15}_{\text{total probability (red or any other colour) king}}}$$

So you can write Bayes theorem also as:

$$\begin{array}{} P(\hat\theta \vert y) &=& \frac{P(y \text{ and } \hat\theta)}{\sum_{\forall \theta_i \in \Theta} P(y \text{ and } \theta_i)} \\ &=& \frac{P(y \text{ and } \hat\theta)}{ P(y \text{ and } \theta_1) + P(y \text{ and } \theta_2) + ... + P(y \text{ and } \theta_{n-1}) + P(y \text{ and } \theta_{n})}\\ &=&\frac{P(y \text{ and } \hat\theta)}{P(y)} \end{array} $$

that is, you comparing a single cell $P(y \text{ and } \hat\theta)$ with a sum of $P(y \text{ and } \theta_i)$ for all possible $\theta_i$ (the sum in a row, which equals the marginal probability $P(y)$).

In addition, you could write

$$P(\hat\theta \vert y) = \frac{P(y \text{ and } \hat\theta)}{P(y)} = \frac{P(y \vert \hat\theta)P(\theta)}{P(y)}$$

you could see this as

  • $P(y \vert \hat\theta)$ is the likelihood. It gives how likely the result/observation $y$ is given the theory $\theta$. The higher this number relatively is, the more the particular observation is support for the particular theory.
  • $P(\theta)$ this is the prior probability of $\theta$ in some cases this is not really known and people call it prior believes. The likelihood should not be considered in isolation. Some high likelihood is "proof" for a given theory, but one must also consider it in conjunction with the probaility of the theory. In your question, you write that $P(y \vert \theta)$ can be thought of a surface, but that is not a surface depicting joint probability. You would need to scale the different lines, $P(y \vert \theta_i)$, in the surface for each $\theta_i$ according to the probability $P(\theta_i)$
  • $P(y)$ you could see this as a normalization constant. For example: if you look in the table te first row, the 1/60, 1/60, 1/15, 1/15, are joint probabilities for kings of hearts, kings of diamonds, kings of clubs and kings of spades. They give the relative conditional probability for hearts, diamonds, clubs and spades, if the card is a king. But, you can not just say that 1/60 is the probability for hearts given that the card is king. You first need to go from relative conditional probabilities to absolute conditional probabilities by scaling it by the sum 1/60+1/60+1/15+1/15.
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.