25
$\begingroup$

(I'm a newbie at stats. I'm a mathematician and a programmer and I'm trying to build something like a naive Bayesian spam filter.)

I've noticed in many places that people tend to break down the denominator in the equation from Bayes' Theorem. So instead of this:

$\frac{P(A|B)\cdot P(B)}{P(A)}$

We are presented with this:

$\frac{P(A|B)\cdot P(B)}{P(A|B)\cdot P(B)+P(A|\neg B)\cdot P(\neg B)}$

You can see that this convention is used in this Wikipedia article and in this insightful post by Tim Peters.

I am baffled by this. Why is the denominator broken down like this? How does that help things at all? What's so complicated about calculating $P(A)$, which in the case of spam filters would be The probability that the word "cheese" appears in an email, regardless of whether it's spam or not?

$\endgroup$
3
  • $\begingroup$ I suspect that the answer is domain specific (i.e., specific to spam filters). If you can calculate the components P(A|B) etc then you should be able to calculate the simpler P(A) as you stated. Or, perhaps the answer is related to pedagogy so that readers understand the relationship between P(A) and its decomposition in terms of P(A|B), P(B) etc. $\endgroup$ – user28 Oct 12 '10 at 14:06
  • 1
    $\begingroup$ I don't have a strong answer, but I can say that I've made made stupid mistakes on tests where I could've simply plugged givens into the explicit denominator, but I thought I knew P(A) and I was wrong. $\endgroup$ – Wayne Oct 5 '11 at 21:36
  • $\begingroup$ For future viewers of this Q: This 3b1b video clearly and intuitively explains the reason for the summation in the denominator of Baye's. While the explanation was clear, I still had to watch it twice, so if you don't get it at first, watch a second time and follow along on paper: youtube.com/watch?v=HZGCoVF3YvM. $\endgroup$ – rocksNwaves Mar 4 at 15:21
17
$\begingroup$

The short answer to your question is, "most of the time we don't know what P(cheese) is, and it is often (relatively) difficult to calculate."

The longer answer why Bayes' Rule/Theorem is normally stated in the way that you wrote is because in Bayesian problems we have - sitting in our lap - a prior distribution (the P(B) above) and likelihood (the P(A|B), P(A|notB) above) and it is a relatively simple matter of multiplication to compute the posterior (the P(B|A)). Going to the trouble to reexpress P(A) in its summarized form is effort that could be spent elsewhere.

It might not seem so complicated in the context of an email because, as you rightly noted, it's just P(cheese), right? The trouble is that with more involved on-the-battlefield Bayesian problems the denominator is an unsightly integral, which may or may not have a closed-form solution. In fact, sometimes we need sophisticated Monte Carlo methods just to approximate the integral and churning the numbers can be a real pain in the rear.

But more to the point, we usually don't even care what P(cheese) is. Bear in mind, we are trying to hone our belief regarding whether or not an email is spam, and couldn't care less about the marginal distribution of the data (the P(A), above). It is just a normalization constant, anyway, which doesn't depend on the parameter; the act of summation washes out whatever info we had about the parameter. The constant is a nuisance to calculate and is ultimately irrelevant when it comes to zeroing in on our beliefs about whether or not the email's spam. Sometimes we are obliged to calculate it, in which case the quickest way to do so is with the info we already have: the prior and likelihood.

$\endgroup$
1
  • 2
    $\begingroup$ Could someone provide and example of 'an unsightly integral, which may or may not have a closed-form solution', which would be used in some problem? $\endgroup$ – PaulG Oct 29 '17 at 0:02
9
$\begingroup$

One reason for using the total probability rule is that we often deal with the component probabilities in that expression and it's straightforward to find the marginal probability by simply plugging in the values. For an illustration of this, see the following example on Wikipedia:

Another reason is recognizing equivalent forms of Bayes' Rule by manipulating that expression. For example:

$P(B|A) = \frac{P(A|B) P(B)}{P(A|B)P(B) + P(A|\lnot B)P(\lnot B)}$

Divide through the RHS by the numerator:

$P(B|A) = \frac{1} {1 + \frac{P(A|\lnot B)}{P(A|B)} \frac{P(\lnot B)}{P(B)}}$

Which is a nice equivalent form for Bayes' Rule, made even handier by subtracting this from the original expression to obtain:

$\frac{P(\lnot B|A)}{P(B|A)} = \frac{P(A|\lnot B)} {P(A|B)} \frac {P(\lnot B)} {P(B)}$

This is Bayes' Rule stated in terms of Odds, i.e. posterior odds against B = Bayes factor against B times the prior odds against B. (Or you could invert it to get an expression in terms of odds for B.) The Bayes factor is the ratio of the likelihoods of your models. Given that we're uncertain about the underlying data generating mechanism, we observe data and update our beliefs.

I'm not sure if you find this useful, but hopefully it's not baffling; you should obviously work with the expression that works best for your scenario. Maybe someone else can pipe in with even better reasons.

$\endgroup$
1
  • $\begingroup$ You can go one step further and take logs. Then you have log-posterior ratio = log-prior ratio + log-likelihood ratio $\endgroup$ – probabilityislogic Mar 1 '11 at 8:43
7
$\begingroup$

Previous replies are detailed enough, but an intuitive way of looking why $P (A) $ (ie dinominator in the Bayes theorem) is broken into two cases.

It is hard to comment about what is the $P(A)$ without any knowledge whether the email is ham or spam. You are correct that "cheese" appears in spam as well as in ham, but if you look at the probability of appearance of "cheese" given that the email is ham ($P(A | B)$, $B$ is representative for ham), you can definitely say a lot of it. At least in my case, I don't receive a lot of spams which contain cheese, therefore in my case $P(A | B)$ will be high (say 90%). Similarly, $P(A | \neg B)$ will be low in my case, as not a lot of spams contain the word cheese. Basically, we try to look at the occurrence of event of interest (here A) partitioned into two disjoint events, $B$ and $\neg B$. If we partition A into two separate events, we can say better about the conditional probabilities $P(A | B)$ and $P(A | \neg B)$. In order to get the total probability we also need to weight for the conditional probabilities for the occurance of events on which we condition ie $P(B)$ and $P(\neg B)$. Therefore the final expression $$P(A) = P(A|B)\cdot P(B)+P(A|\neg B)\cdot P(\neg B)$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.