Bayes' Theorem Intuition

Question

I've been trying to develop an intuition based understanding of Bayes' theorem in terms of the prior, posterior, likelihood and marginal probability. For that I use the following equation: $$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$$ where $A$ represents a hypothesis or belief and $B$ represents data or evidence.
I've understood the concept of the posterior - it's a unifying entity that combines the prior belief and the likelihood of an event. What I don't understand is what does the likelihood signify? And why is the marginal probability in the denominator?
After reviewing a couple of resources I came across this quote:

The likelihood is the weight of event $B$ given by the occurrence of $A$ ... $P(B|A)$ is the posterior probability of event $B$ , given that event $A$ has occurred.

The above 2 statements seem identical to me, just written in different ways. Can anyone please explain the difference between the two?

Answer 1

Although there are four components listed in Bayes' law, I prefer to think in terms of three conceptual components:
$$ \underbrace{P(B|A)}_2 = \underbrace{\frac{P(A|B)}{P(A)}}_3 \underbrace{P(B)}_1 $$

1. The prior is what you believed about $B$ before having encountered a new and relevant piece of information (i.e., $A$).
2. The posterior is what you believe (or ought to, if you are rational) about $B$ after having encountered a new and relevant piece of information.
3. The quotient of the likelihood divided by the marginal probability of the new piece of information indexes the informativeness of the new information for your beliefs about $B$.

Answer 2

There are several good answers already, but perhaps this can add something new ...

I always think of Bayes rule in terms of the component probabilities, which can be understood geometrically in terms of the events $A$ and $B$ as pictured below.

The marginal probabilities $P(A)$ and $P(B)$ are given by the areas of the corresponding circles. All possible outcomes are represented by $P(A \cup B)=1$, corresponding to the set of events "$A$ or $B$". The joint probability $P(A \cap B)$ corresponds to the event "$A$ and $B$".

In this framework, the conditional probabilities in Bayes theorem can be understood as ratios of areas. The probability of $A$ given $B$ is the fraction of $B$ occupied by $A \cap B$, expressed as $$P(A\vert B)=\frac{P(A \cap B)}{P(B)}$$ Similarly, the probability of $B$ given $A$ is the fraction of $A$ occupied by $A \cap B$, i.e. $$P(B\vert A)=\frac{P(A \cap B)}{P(A)}$$

Bayes theorem is really just a mathematical consequence of the above definitions, which can be restated as $$P(B\vert A)P(A)=P(A \cap B)=P(A\vert B)P(B)$$ I find this symmetric form of Bayes theorem to be much easier to remember. That is, the identity holds regardless of which $p(A)$ or $p(B)$ is labelled "prior" vs. "posterior".

(Another way of understanding the above discussion is given in my answer to this question, from a more "accounting spreadsheet" point of view.)

Answer 3

@gung has a great answer. I would add one example to explain the "initiation" in a real world example.

For better connection with real world examples, I would like to change the notation, where use $H$ to represent the hypothesis (the $A$ in your equation), and use $E$ to represent evidence. (the $B$ in your equation.)

So the formula is

$$P(H|E) = \frac{P(E|H)P(H)}{P(E)}$$

Note the same formula can be written as

$$P(H|E) \propto {P(E|H)P(H)}$$

where $\propto$ means proportional to and $P(E|H)$ is the likelihood and $P(H)$ is the prior. This equation means that the posterior will be larger, if the right side of the equation larger. And you can think about $P(E)$ is a normalization constant to make the number into probability (the reason I say it is a constant is because the evidence $E$ is already given.).

For a real world example, suppose we are doing some fraud detection on credit card transactions. Then the hypothesis would be $H \in \{0,1\}$ where represent the transaction is a normal or fraudulent. (I picked extreme imbalanced case to show the intuition).

From domain knowledge, we know most transactions would be normal, only very few are fraud. Let us assume an expert told us there are $1$ in $1000$ would be fraud. So we can say the prior is $P(H=1)=0.001$, and $P(H=0)=0.999$.

The ultimate goal is calculating $P(H|E)$ which means we want to know if a transaction is a fraud not not based on the evidence in addition to prior. If you look at the right side of the equation, we decompose it into likelihood and prior.

Where we already explained what is prior, here we explain what is likelihood. Suppose we have two types of evidence, $E\in\{0,1\}$ that represent, if we are seeing normal or strange geographical location of the transaction.

The likelihood $P(E=1|H=0)$ may be small, which means given a normal transaction, it is very unlikely the location is strange. On the other hand, $P(E=1|H=1)$ can be large.

Suppose, we observed $E=1$ we want to see if it is a fraud or not, we need to consider both prior and likelihood. Intuitively, from prior, we know there are very few fraud transactions, we would likely to be very conservative to make a fraud classification, unless the evidence is very strong. Therefore, the product between two will consider two factors at same time.

Answer 4

Aug 7 2015 Medium article explains with many pictures! 1 in 10 people are sick.

To simplify the example, we assume we know which ones are sick and which ones are healthy, but in a real test you don’t know that information. Now we test everybody for the disease:

The true positives = number of positive results among the sick population = #(Positive | Sick) = 9.

Now the interesting question, what's probability of being sick if you test positive? In math, $\Pr(Sick | Positive)?$

Answer 5

Here's a super straightforward and intuitive explanation.

https://metagrokker.medium.com/an-intuitive-interpretation-of-bayes-theorem-d3e43b05bb2a

Here's a summary; think of Bayes in the following form:

P(A|B) = P(A) * (P(B|A) / P(B))

P(A) is our confidence about hypothesis A being true, before accounting for the new evidence B. The equation multiplies P(A) by the ratio (P(B|A) / P(B)) to turn P(A) into the posterior probability P(A|B); our 'updated' confidence about hypothesis A being true, after accounting for evidence B. This is what Bayes does, it updates our knowledge based on new evidence, turning our prior knowledge to posterior.

Now, what does the ratio mean? P(B) is the probability of our observation happening, while P(B|A) is the probability of the observation happening if A were true. Therefore, this ratio captures how much more likely observing B becomes when A is true vs generally. If A being true results in B being more likely, this means B is a good sign that A is true, so P(B|A) will be greater than P(B), making the ratio greater than one, and therefore it will increase our confidence on A. So the ratio captures how significant observing B is to A being true, and the equation adjusts our knowledge proportionally to this.

Answer 6

Here is an additional graphic to the intuition of Bayes rule in terms $A$ and $B$ following some joint distribution.

$$P(B|A)\cdot P(A) = P(A,B) = P(A|B) \cdot P(B)$$

or

$$P(B|A) = \frac{P(A,B)}{P(A)} = \frac{P(A|B) \cdot P(B)}{P(A) }$$

The prior (independent of the observation of A), is the marginal distribution in this joint distribution. The posterior (dependent on the observation of A), is the conditional distribution in this joint distribution.

The view is often ${P(A|B) \cdot P(B)}$ instead of ${P(A,B)}$. This is, I believe, because often $B$ is some model parameter and $A$ is an observation and we can describe $P(A|B)$ well in terms of some theoretical model for the observation as function of the coefficient.

Answer 7

Note that Bayes' rule is

$P(a|b)=\frac{P(b,a)}{P(b)}=\frac{P(b,a)}{P(b)P(a)}P(a)$.

Note the ratio

$$\frac{P(b,a)}{P(b)P(a)}.$$

If $B \perp A$, then $P(b,a)=P(b)P(a)$. So it’s almost like telling us how far the joint deviates from full independence, or how much information the variables have in common.

Of course this is also evident from the ration $\frac{P(b|a)}{P(b)},$ but the symmetry of the expression above is nice, and more interestingly, the log of this $\frac{P(b,a)}{P(b)P(a)}$ is the pointwise mutual information!!!

The full mutual information is:

$$I(A|B) = \sum_{a,b}P_{}(a,b)\mathbf{\log\frac{P_{}(b,a)}{P(b)P(a)}}.$$

Answer 8

I often find viewing the theorem as a table, with the possible outcomes for "B" as the rows, and the possible outcomes for "A" as the columns. The joint probabilities $P(A,B)$ are the values for each cell. In this table we have

likelihood = row proportions posterior = column proportions

The prior and marginal are analogously defined, but based on "totals" instead of a particular column

marginal = row total proportions prior = column total proportions

I find this helps me.