I've been trying to develop an intuition based understanding of Bayes' theorem in terms of the prior, posterior, likelihood and marginal probability. For that I use the following equation: $$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$$ where $A$ represents a hypothesis or belief and $B$ represents data or evidence.
I've understood the concept of the posterior - it's a unifying entity that combines the prior belief and the likelihood of an event. What I don't understand is what does the likelihood signify? And why is the marginal probability in the denominator?
After reviewing a couple of resources I came across this quote:

The likelihood is the weight of event $B$ given by the occurrence of $A$ ... $P(B|A)$ is the posterior probability of event $B$ , given that event $A$ has occurred.

The above 2 statements seem identical to me, just written in different ways. Can anyone please explain the difference between the two?

  • 6
    $\begingroup$ You have a typo (or a misconception). $B$ should be the "hypothesis or belief", and $A$ should be the "data or evidence" in your formulation. $\endgroup$ – gung - Reinstate Monica Oct 7 '16 at 17:36
  • 2
    $\begingroup$ see my answer at math.stackexchange.com/a/1943255/1505 that is how I ended up understanding it intuitively $\endgroup$ – Lyndon White Oct 8 '16 at 4:00

Although there are four components listed in Bayes' law, I prefer to think in terms of three conceptual components:
$$ \underbrace{P(B|A)}_2 = \underbrace{\frac{P(A|B)}{P(A)}}_3 \underbrace{P(B)}_1 $$

  1. The prior is what you believed about $B$ before having encountered a new and relevant piece of information (i.e., $A$).
  2. The posterior is what you believe (or ought to, if you are rational) about $B$ after having encountered a new and relevant piece of information.
  3. The quotient of the likelihood divided by the marginal probability of the new piece of information indexes the informativeness of the new information for your beliefs about $B$.
| cite | improve this answer | |

There are several good answers already, but perhaps this can add something new ...

I always think of Bayes rule in terms of the component probabilities, which can be understood geometrically in terms of the events $A$ and $B$ as pictured below.

Event Sets

The marginal probabilities $P(A)$ and $P(B)$ are given by the areas of the corresponding circles. All possible outcomes are represented by $P(A \cup B)=1$, corresponding to the set of events "$A$ or $B$". The joint probability $P(A \cap B)$ corresponds to the event "$A$ and $B$".

In this framework, the conditional probabilities in Bayes theorem can be understood as ratios of areas. The probability of $A$ given $B$ is the fraction of $B$ occupied by $A \cap B$, expressed as $$P(A\vert B)=\frac{P(A \cap B)}{P(B)}$$ Similarly, the probability of $B$ given $A$ is the fraction of $A$ occupied by $A \cap B$, i.e. $$P(B\vert A)=\frac{P(A \cap B)}{P(A)}$$

Bayes theorem is really just a mathematical consequence of the above definitions, which can be restated as $$P(B\vert A)P(A)=P(A \cap B)=P(A\vert B)P(B)$$ I find this symmetric form of Bayes theorem to be much easier to remember. That is, the identity holds regardless of which $p(A)$ or $p(B)$ is labelled "prior" vs. "posterior".

(Another way of understanding the above discussion is given in my answer to this question, from a more "accounting spreadsheet" point of view.)

| cite | improve this answer | |

@gung has a great answer. I would add one example to explain the "initiation" in a real world example.

For better connection with real world examples, I would like to change the notation, where use $H$ to represent the hypothesis (the $A$ in your equation), and use $E$ to represent evidence. (the $B$ in your equation.)

So the formula is

$$P(H|E) = \frac{P(E|H)P(H)}{P(E)}$$

Note the same formula can be written as

$$P(H|E) \propto {P(E|H)P(H)}$$

where $\propto$ means proportional to and $P(E|H)$ is the likelihood and $P(H)$ is the prior. This equation means that the posterior will be larger, if the right side of the equation larger. And you can think about $P(E)$ is a normalization constant to make the number into probability (the reason I say it is a constant is because the evidence $E$ is already given.).

For a real world example, suppose we are doing some fraud detection on credit card transactions. Then the hypothesis would be $H \in \{0,1\}$ where represent the transaction is a normal or fraudulent. (I picked extreme imbalanced case to show the intuition).

From domain knowledge, we know most transactions would be normal, only very few are fraud. Let us assume an expert told us there are $1$ in $1000$ would be fraud. So we can say the prior is $P(H=1)=0.001$, and $P(H=0)=0.999$.

The ultimate goal is calculating $P(H|E)$ which means we want to know if a transaction is a fraud not not based on the evidence in addition to prior. If you look at the right side of the equation, we decompose it into likelihood and prior.

Where we already explained what is prior, here we explain what is likelihood. Suppose we have two types of evidence, $E\in\{0,1\}$ that represent, if we are seeing normal or strange geographical location of the transaction.

The likelihood $P(E=1|H=0)$ may be small, which means given a normal transaction, it is very unlikely the location is strange. On the other hand, $P(E=1|H=1)$ can be large.

Suppose, we observed $E=1$ we want to see if it is a fraud or not, we need to consider both prior and likelihood. Intuitively, from prior, we know there are very few fraud transactions, we would likely to be very conservative to make a fraud classification, unless the evidence is very strong. Therefore, the product between two will consider two factors at same time.

| cite | improve this answer | |
  • $\begingroup$ I think there may be a typo in the prior: $P(H=0)$ should be $0.999$ and $P(H=1)=0.001$, right? $\endgroup$ – gc5 Nov 22 '17 at 17:42

Aug 7 2015 Medium article explains with many pictures! 1 in 10 people are sick.

enter image description here

To simplify the example, we assume we know which ones are sick and which ones are healthy, but in a real test you don’t know that information. Now we test everybody for the disease:

enter image description here

The true positives = number of positive results among the sick population = #(Positive | Sick) = 9.

Now the interesting question, what's probability of being sick if you test positive? In math, $\Pr(Sick | Positive)?$

enter image description here

| cite | improve this answer | |

Note that Bayes' rule is


Note the ratio


If $B \perp A$, then $P(b,a)=P(b)P(a)$. So it’s almost like telling us how far the joint deviates from full independence, or how much information the variables have in common.

Interestingly, the log of this ratio is also present in mutual information:

$$I(A|B) = \sum_{a,b}P_{}(a,b)\mathbf{\log\frac{P_{}(b,a)}{P(b)P(a)}}$$

| cite | improve this answer | |

I often find viewing the theorem as a table, with the possible outcomes for "B" as the rows, and the possible outcomes for "A" as the columns. The joint probabilities $P(A,B)$ are the values for each cell. In this table we have

likelihood = row proportions posterior = column proportions

The prior and marginal are analogously defined, but based on "totals" instead of a particular column

marginal = row total proportions prior = column total proportions

I find this helps me.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.