15
$\begingroup$

I know the Bayes rule is derived from the conditional probability. But intuitively, what is the difference? The equation looks the same to me. The nominator is the joint probability and the denominator is the probability of the given outcome.

This is the conditional probability: $P(A∣B)=\frac{P(A \cap B)}{P(B)}$

This is the Bayes' rule: $P(A∣B)=\frac{P(B|A) * P(A)}{P(B)}$.

Isn't $P(B|A)*P(A)$ and $P(A \cap B)$ the same? When $A$ and $B$ are independent, there is no need to use the Bayes rule, right?

$\endgroup$
  • 2
    $\begingroup$ If you would add the specific equations that look the same to you to your question, someone might be able to help you. The two that I am familiar with look quite different to me but there is a long tradition on stats.SE to say that Bayes formula is $$P(A\mid B) = \frac{P(A\cap B)}{P(B)}$$ which is actually the definition of the conditional probability of $A$ given $B$, and not Bayes formula at all. $\endgroup$ – Dilip Sarwate Dec 9 '16 at 2:59
  • $\begingroup$ @DilipSarwate, I have updated my question. $\endgroup$ – News_is_Selection_Bias Dec 9 '16 at 18:36
  • $\begingroup$ To your final question: yes these are the same! That doesn't mean Bayes' rule isn't a useful formula, however. The conditional probability formula doesn't give us the probability of A given B. Semantically, I'd say there's always a need to use Bayes' rule, but when A and B are independent the rule can be reduced to a much simpler form. $\endgroup$ – Jacob Socolar Dec 9 '16 at 19:03
  • $\begingroup$ I understand Bayes rule is useful. Given A and B are not independent, what the difference of conditional probability function and Bayes rule if the nominators are basically the same( correct me if i am wrong)? $\endgroup$ – News_is_Selection_Bias Dec 9 '16 at 19:11
  • $\begingroup$ My answer here provides another view of essentially this issue. $\endgroup$ – GeoMatt22 Dec 10 '16 at 5:08
23
$\begingroup$

OK, now that you have updated your question to include the two formulas:

$$P(A\mid B) = \frac{P(A\cap B)}{P(B)} ~~ \text{provided that } P(B) > 0, \tag{1}$$ is the definition of the conditional probability of $A$ given that $B$ occurred. Similarly, $$P(B\mid A) = \frac{P(B\cap A)}{P(A)} = \frac{P(A\cap B)}{P(A)} ~~ \text{provided that } P(A) > 0, \tag{2}$$ is the definition of the conditional probability of $B$ given that $A$ occurred. Now, it is true that it is a trivial matter to substitute the value of $P(A\cap B)$ from $(2)$ into $(1)$ to arrive at $$P(A\mid B) = \frac{P(B\mid A)P(A)}{P(B)} ~~ \text{provided that } P(A), P(B) > 0, \tag{3}$$ which is Bayes' formula but notice that Bayes's formula actually connects two different conditional probabilities $P(A\mid B)$ and $P(B\mid A)$, and is essentially a formula for "turning the conditioning around". The Reverend Thomas Bayes referred to this in terms of "inverse probability" and even today, there is vigorous debate as to whether statistical inference should be based on $P(B\mid A)$ or the inverse probability (called the a posteriori or posterior probability).

It is undoubtedly as galling to you as it was to me when I first discovered that Bayes' formula was just a trivial substitution of $(2)$ into $(1)$. Perhaps if you have been born 250 years ago, you (Note: the OP masqueraded under username AlphaBetaGamma when I wrote this answer but has since changed his username) could have made the substitution and then people today would be talking about the AlphaBetaGamma formula and the AlphaBetaGammian heresy and the Naive AlphaBetaGamma method$^*$ instead of invoking Bayes' name everywhere. So let me console you on your loss of fame by pointing out a different version of Bayes' formula. The Law of Total Probability says that $$P(B) = P(B\mid A)P(A) + P(B\mid A^c)P(A^c) \tag{4}$$ and using this, we can write $(3)$ as

$$P(A\mid B) = \frac{P(B\mid A)P(A)}{P(B\mid A)P(A) + P(B\mid A^c)P(A^c)}, \tag{5}$$ or more generally as $$P(A_i\mid B) = \frac{P(B\mid A_i)P(A_i)}{P(B\mid A_1)P(A_1) + P(B\mid A_2)P(A_2) + \cdots + P(B\mid A_n)P(A_n)}, \tag{6}$$ where the posterior probability of a possible "cause" $A_i$ of a "datum" $B$ is related to $P(B\mid A_i)$, the likelihood of the observation $B$ when $A_i$ is the true hypothesis and $P(A_i)$, the prior probability (horrors!) of the hypothesis $A_i$.


$^*$ There is a famous paper R. Alpher, H. Bethe, and G. Gamow, "The Origin of Chemical Elements", Physical Review, April 1, 1948, that is commonly referred to as the $\alpha\beta\gamma$ paper.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Hello Sir, Could you please explain what you mean by 'turning the conditioning around'? $\endgroup$ – formatkaka Feb 18 '17 at 17:53
  • 1
    $\begingroup$ @Siddhant Going from $P(A\mid B)$ to $P(B\mid A)$ is what I mean by "turning the conditioning around". Please ignore the phrase, which I made up on the spot to give a name to what Bayes' Theorem does (it gives an expression for $P(A\mid B)$ in terms of $P(B\mid A)$) since it confuses you so much. $\endgroup$ – Dilip Sarwate Feb 18 '17 at 20:52
2
$\begingroup$

One way to intuitively think of Bayes' theorem is that when any one of these is easy to calculate

$$P(A∣B) ~~ \text{or } P(B∣A)$$

we can calculate the other one even though the other one seems to be bit hard at first

Consider an example, Here $$P(A∣B)$$ is say I have a curtain and I told you there is an animal behind the curtain and given it is a four legged animal what is the probability of that animal being dog ?

It is hard to find a probability for that.

But you can find the answer for $$P(B∣A)$$ What is the probability of a four legged animal behind the curtain and given that it is a dog, now it is easy to calculate 
it could be nearly 1 and you plug in those values in the bayes theorem and you ll find the answer for $$P(A∣B)$$ that is the probability of the animal being a dog which at first was hard.

Now this is just an over simplified version where you can intuitively think why rearranging the formula could help us. I hope this helps.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.