20
$\begingroup$

I've been trying to develop an intuition based understanding of Bayes' theorem in terms of the prior, posterior, likelihood and marginal probability. For that I use the following equation: $$P(B|A) = \frac{P(A|B)P(B)}{P(A)}$$ where $A$ represents a hypothesis or belief and $B$ represents data or evidence.
I've understood the concept of the posterior - it's a unifying entity that combines the prior belief and the likelihood of an event. What I don't understand is what does the likelihood signify? And why is the marginal probability in the denominator?
After reviewing a couple of resources I came across this quote:

The likelihood is the weight of event $B$ given by the occurrence of $A$ ... $P(B|A)$ is the posterior probability of event $B$ , given that event $A$ has occurred.

The above 2 statements seem identical to me, just written in different ways. Can anyone please explain the difference between the two?

$\endgroup$
  • 4
    $\begingroup$ You have a typo (or a misconception). $B$ should be the "hypothesis or belief", and $A$ should be the "data or evidence" in your formulation. $\endgroup$ – gung - Reinstate Monica Oct 7 '16 at 17:36
  • 1
    $\begingroup$ see my answer at math.stackexchange.com/a/1943255/1505 that is how I ended up understanding it intuitively $\endgroup$ – Lyndon White Oct 8 '16 at 4:00
25
$\begingroup$

Although there are four components listed in Bayes' law, I prefer to think in terms of three conceptual components:
$$ \underbrace{P(B|A)}_2 = \underbrace{\frac{P(A|B)}{P(A)}}_3 \underbrace{P(B)}_1 $$

  1. The prior is what you believed about $B$ before having encountered a new and relevant piece of information (i.e., $A$).
  2. The posterior is what you believe (or ought to, if you are rational) about $B$ after having encountered a new and relevant piece of information.
  3. The quotient of the likelihood divided by the marginal probability of the new piece of information indexes the informativeness of the new information for your beliefs about $B$.
$\endgroup$
17
$\begingroup$

There are several good answers already, but perhaps this can add something new ...

I always think of Bayes rule in terms of the component probabilities, which can be understood geometrically in terms of the events $A$ and $B$ as pictured below.

Event Sets

The marginal probabilities $P(A)$ and $P(B)$ are given by the areas of the corresponding circles. All possible outcomes are represented by $P(A \cup B)=1$, corresponding to the set of events "$A$ or $B$". The joint probability $P(A \cap B)$ corresponds to the event "$A$ and $B$".

In this framework, the conditional probabilities in Bayes theorem can be understood as ratios of areas. The probability of $A$ given $B$ is the fraction of $B$ occupied by $A \cap B$, expressed as $$P(A\vert B)=\frac{P(A \cap B)}{P(B)}$$ Similarly, the probability of $B$ given $A$ is the fraction of $A$ occupied by $A \cap B$, i.e. $$P(B\vert A)=\frac{P(A \cap B)}{P(A)}$$

Bayes theorem is really just a mathematical consequence of the above definitions, which can be restated as $$P(B\vert A)P(A)=P(A \cap B)=P(A\vert B)P(B)$$ I find this symmetric form of Bayes theorem to be much easier to remember. That is, the identity holds regardless of which $p(A)$ or $p(B)$ is labelled "prior" vs. "posterior".

(Another way of understanding the above discussion is given in my answer to this question, from a more "accounting spreadsheet" point of view.)

$\endgroup$
8
$\begingroup$

@gung has a great answer. I would add one example to explain the "initiation" in a real world example.

For better connection with real world examples, I would like to change the notation, where use $H$ to represent the hypothesis (the $A$ in your equation), and use $E$ to represent evidence. (the $B$ in your equation.)

So the formula is

$$P(H|E) = \frac{P(E|H)P(H)}{P(E)}$$

Note the same formula can be written as

$$P(H|E) \propto {P(E|H)P(H)}$$

where $\propto$ means proportional to and $P(E|H)$ is the likelihood and $P(H)$ is the prior. This equation means that the posterior will be larger, if the right side of the equation larger. And you can think about $P(E)$ is a normalization constant to make the number into probability (the reason I say it is a constant is because the evidence $E$ is already given.).

For a real world example, suppose we are doing some fraud detection on credit card transactions. Then the hypothesis would be $H \in \{0,1\}$ where represent the transaction is a normal or fraudulent. (I picked extreme imbalanced case to show the intuition).

From domain knowledge, we know most transactions would be normal, only very few are fraud. Let us assume an expert told us there are $1$ in $1000$ would be fraud. So we can say the prior is $P(H=1)=0.001$, and $P(H=0)=0.999$.

The ultimate goal is calculating $P(H|E)$ which means we want to know if a transaction is a fraud not not based on the evidence in addition to prior. If you look at the right side of the equation, we decompose it into likelihood and prior.

Where we already explained what is prior, here we explain what is likelihood. Suppose we have two types of evidence, $E\in\{0,1\}$ that represent, if we are seeing normal or strange geographical location of the transaction.

The likelihood $P(E=1|H=0)$ may be small, which means given a normal transaction, it is very unlikely the location is strange. On the other hand, $P(E=1|H=1)$ can be large.

Suppose, we observed $E=1$ we want to see if it is a fraud or not, we need to consider both prior and likelihood. Intuitively, from prior, we know there are very few fraud transactions, we would likely to be very conservative to make a fraud classification, unless the evidence is very strong. Therefore, the product between two will consider two factors at same time.

$\endgroup$
  • $\begingroup$ I think there may be a typo in the prior: $P(H=0)$ should be $0.999$ and $P(H=1)=0.001$, right? $\endgroup$ – gc5 Nov 22 '17 at 17:42
0
$\begingroup$

Note that Bayes' rule is

$$P(A|B)=\frac{P(B|A)P(A)}{P(B)}=\frac{P(B,A)}{P(B)P(A)}P(A)$$

Note the term

$$\frac{P(B,A)}{P(B)P(A)}$$.

This term is the same whether we are using Bayes' rule to write $P(A|B)$ as above $or$

$$P(B|A)=\frac{P(A|B)P(B)}{P(A)}=\frac{P(A,B)}{P(A)P(B)}P(B)=\frac{P(B,A)}{P(B)P(A)}P(B)$$

Hence this term captures information about the relatedness of the two variables in a way that is agnostic to the choice of what goes where in the conditional probability on the left hand side of the equation. This is odd because the likelihood divided by the marginal (say, in the first equation where it is $\frac{P(B|A)}{P(B)}$) does not seem to have this symmetry at first glance.

So Bayes' rule adjusts the prior $P(B)$, which would equal $P(B|A)$ were $A \perp B$, by some measure of the amount of dependence between $A$ and $B$.

This measure is also present in mutual information

$$I(A|B) = \sum_{a,b}P_{AB}(a,b)\mathbf{\log\frac{P_{AB}(a,b)}{P_A(a)P_B(b)}}$$

The following also holds just using $\frac{P(B|A)}{P(B)}$, but it is interesting to note using $\frac{P(B,A)}{P(B)P(A)}$:

If $B \perp A$, then $P(B,A)=P(B)P(A)$ and so $\frac{P(B,A)}{P(B)P(A)}=1$.

For the case where $A \not\perp B$, for $P(A|B)$, conditioning on $B$ can either increase or decrease the prior $P(A)$. When it increases the prior, $P(A,B) \gt P(A)P(B)$ --- the variables occur jointly more often than would be expected were they independent. When it decreases, $P(A,B) \lt P(A)P(B)$ --- the opposite is true.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.