3
$\begingroup$

Let's say we have a problem of predicting whether a storm is coming or not. We have a model P(storm is coming | how many clouds are outside), and have another model P(storm is coming | how scared the dogs are).

My question in general is how to combine these two models in a sensible way. It seems to make sense if I just multiply the two together - however, I don't know what this expression evaluates to:

P(storm is coming | how many clouds are outside) * P(storm is coming | how scared the dogs are) = ?

Or, with simpler notation:

P(A|B)P(A|C) = ? (where B and C are independent)

I have a feeling the answer is P(A|B)P(A|C) = P(A|B,C) but I can't prove it. (As many pointed out, this is wrong)

Edit: Sorry for the confusion, the question here is "what is P(A|B)P(A|C) = ?"

$\endgroup$
  • $\begingroup$ Sorry that is not true. $\endgroup$ – Michael R. Chernick Jul 1 '17 at 1:42
  • $\begingroup$ @MichaelChernick I'm glad you say that is not true. What is true then? Does P(A|B)P(A|C) simplify to anything? $\endgroup$ – foobar Jul 1 '17 at 1:50
  • 1
    $\begingroup$ Given the text your question seems to be what is the $f$ that make the following true $P(A|B,C)=f(P(A|B),P(A|C))$? $\endgroup$ – Peter Mølgaard Pallesen Jul 9 '17 at 15:23
  • 3
    $\begingroup$ I agree with @PeterMølgaardPallesen. The edit says that you're interested in simplifying P(A|B)P(A|C), but doing so would NOT AT ALL be a useful way to predict the probability that a storm is coming in your original example. Some trivial examples show this. For example, imagine P(A|B) = 1 and P(A|C) = 0.5. Then the multiplication gives .5, but if you observe B then the probability of A is still 1, regardless of whether C is true. Furthermore, note that if B and C both provide information about A, it is unlikely in general that B and C will be independent, as specified in the question. $\endgroup$ – Jacob Socolar Jul 9 '17 at 17:01
  • $\begingroup$ I think you should look at naive Bayes classifiers which are i think what you are looking for en.m.wikipedia.org/wiki/Naive_Bayes_classifier $\endgroup$ – seanv507 Jul 9 '17 at 22:32
4
+50
$\begingroup$

Let's say we have a problem of predicting whether a storm is coming or not.

So we'd like to predict whether a storm is coming or not (event $A$), and we have some clues available to us, namely the amount of clouds in the sky (event $B$) and how scared your dogs are (event $C$).

We can visualise the problem at hand using a Venn diagram:

Venn diagram of events "storm" (A), "cloudy sky" (B) and "scared dogs" (C)

We are interested in calculating the probability of a storm given the clues, $P(A|B,C)$. That quantity isn't represented directly in the diagram; instead, we can get $P(A,B,C)$ (a.k.a $P(A \cap B \cap C)$) from the white central area in the diagram. Fortunately, the relationship between $P(A|B,C)$ and $P(A,B,C)$ is simple:

$$P(A,B,C) = P(A|B,C) \cdot P(B,C)$$

where $P(B,C)$ corresponds to the conjunction between the magenta and white areas of the diagram.

We have a model P(storm is coming | how many clouds are outside), and have another model P(storm is coming | how scared the dogs are)

So we know $P(A|B)$ and $P(A|C)$. Like before, these two quantities are not represented directly in the diagram. Instead, we have $P(A,B)$, which corresponds to the yellow and white areas, and $P(A,C)$, which corresponds to the cyan and white areas. As before, we know the relationship between $P(A,B)$ and $P(A|B)$:

$$P(A,B) = P(A|B) \cdot P(B)$$

Same goes for $P(A,C)$ and $P(A|C)$.

To recap, we would like to know $P(A|B,C)$, which is related to the white area in the Venn diagram. So what happens if we add $P(A)$, $P(B)$ and $P(C)$? We are counting the magenta, yellow and cyan areas twice each, and the white central area three times. So we subtract the magenta, yellow and cyan areas once:

$$P(A) + P(B) + P(C) - P(A,B) - P(A,C) - P(B,C)$$

Except now we removed the white area from the summation; we added the white area three times when we summed up $A$, $B$, and $C$, but we removed it three times when we subtracted $(A,B)$, $(A,C)$ and $(B,C)$. So we add it back:

$$P(A) + P(B) + P(C) - P(A,B) - P(A,C) - P(B,C) + P(A,B,C)$$

We didn't account for the area outside all the circles, which corresponds to $P(\tilde{} A, \tilde{} B, \tilde{} C)$, which is the chance that there is no storm AND there are no clouds AND the dogs aren't scared.

$$P(A) + P(B) + P(C) - P(A,B) - P(A,C) - P(B,C) + P(A,B,C) + P(\tilde{} A, \tilde{} B, \tilde{} C) = 1$$

Let's assume that a storm ocurring with a spotless sky is very unlikely; $P(\tilde{} A, \tilde{} B, \tilde{} C) \approx 0$. In that case,

$$P(A) + P(B) + P(C) - P(A,B) - P(A,C) - P(B,C) + P(A,B,C) = 1$$

Let's apply the transformations we saw before:

$\begin{align} P(A|B,C) \cdot P(B,C) &= 1 - [P(A) + P(B) + P(C) - P(A|B) \cdot P(B) - P(A|C) \cdot P(C) - P(B,C)]\\ P(A|B,C) &= \dfrac{1 - P(A) - P(B) - P(C) + P(A|B) \cdot P(B) + P(A|C) \cdot P(C)}{P(B,C)} + 1 \end{align} $

As you can see, you would need more information if you want to calculate the probability of a storm given your clues. Namely:

  1. The probability of a storm in general;
  2. The probability of a cloudy sky in general;
  3. The probability of your dogs being scared in general; and
  4. The probability that your dogs will be scared AND the sky will be cloudy.

If you think about it, numbers 1-3 make sense:

  1. The clues may increase the probability of a storm, but if there aren't many storms to begin with, then the probability of a storm given your clues will still be small (albeit larger than the baseline probability of a storm);
  2. If you live in a typically cloudy area, the amount of clouds in the sky will probably be a poor predictor of a storm (because it's always cloudy, storm or no storm);
  3. Ditto for your dogs being scared.

Number 4 is a bit trickier. If either your dogs or the sky (or both) are perfect predictors of a storm, then there is no need for the other.

Now all this math assumes that your model outputs $P(\mathrm{storm} | \mathrm{clouds})$ ($P(A | B)$) and $P(\mathrm{storm} | \mathrm{scared\ dogs})$ ($P(A|C)$). However, it is typically easier to observe $P(\mathrm{clouds} | \mathrm{storm})$ ($P(B | A)$) and $P(\mathrm{scared\ dogs} | \mathrm{storm})$ ($P(C|A)$). In that case, we must note that

$$P(A,B) = P(A|B) \cdot P(B) = P(B|A) \cdot P(A)$$

so our previous model becomes

$$P(A|B,C) = \dfrac{1 - P(A) - P(B) - P(C) + P(B|A) \cdot P(A) + P(C|A) \cdot P(A)}{P(B,C)} + 1$$

$\endgroup$
5
$\begingroup$

In my opinion your problem is not about that expression but about modelling. You have different clues (clouds, scared dogs) which provide evidence for a forthcoming event (rain). In other words, if I understand you, your question is actually more about: how to combine different clues?.

This question is dealt with in the field of graphical models. Let me refer to the corresponding Wikipedia site, and the references therein. Specially the paper by David Heckerman (A Tutorial on Learning With Bayesian Networks), and the reference to Christopher Bishop's Machine Learning book.

The example you give is very similar to those presented to describe the explain away effect, that is nicely described in this video.

$\endgroup$
  • 1
    $\begingroup$ +1. I'd add to your references this paper by Kruschke and Liddell, which explains the Bayesian framework quite nicely, and this page, which has a collection of articles designed to show how Bayesian Nets are useful. It's maintained by the authors of a great book about Bayesian Nets. $\endgroup$ – LmnICE Jul 12 '17 at 0:20
1
$\begingroup$

Not really a simplification (if that is possible at all) but, if you wish a relation between $P(a \vert b)P(a \vert c)$ and $P(a \vert b,c)$, you could say that

$P(a \vert b)P(a \vert c) = P(a \vert b,c)^2 \frac{P(c \vert b)}{P(c \vert a,b)} \frac{P(b \vert c)}{P(b \vert a,c)}$

I think that this relation is more theoretic and fun than useful. In practical cases you would use the simpler

$P(a \vert b) = P(a \vert b,c) \frac{P(c \vert b)}{P(c \vert a,b)}$

$P(a \vert c) = P(a \vert b,c) \frac{P(b \vert c)}{P(b \vert a,c)}$

one use is maybe that this tells how you can't give a usefull interpretation for $P(a \vert b)P(a \vert c)$. Say $P(\text{cancer} \vert \text{smoking})=0.1$ and $P(\text{cancer} \vert \text{sports})=0.001$ then how these relate to $P(\text{cancer} \vert \text{smoking and sports})^2$ is dependent on the terms in the fractions and it can be much different from $0.0001$ either larger or smaller and depends on the case.

$\endgroup$
0
$\begingroup$

Bayes' Theorem states:

$$P(A|B)=\frac{P(B|A)P(A)}{P(B)}$$

so logically we have:

$$P(A|B)P(A|C)\\ =\frac{P(B|A)P(A)}{P(B)}\frac{P(C|A)P(A)}{P(C)}\\ =\frac{P(B|A)P(C|A)P(A)^2}{P(B)P(C)} $$

As $B$ and $C$ are independent, I think this is as far as this route extends.

$\endgroup$
0
$\begingroup$

As JonMark Perry has already mentioned, Bayes' theorem prohibits that your initial suspicion is true. The rules for conditional probabilities specifically allow for multiplication of probabilities only if conditioning under the same event (either B or C or both simultaneously). To show you a visualistion of the 2 probabilities take a look at this image of a tree diagram.. Within a branch of the tree, you may multiply the probabilities which is interpreted as both events having to happen simultaneously P(A and B) = P(A|B) * P(B) or P(A and C) = P(A|C) * P(C) . In order to combine to two different branches (the directions would be B and C) the probabilities have to be added. P(A|B) * P(B)+ P(A|C) * P(C) is then the probability that the storm comes up, independent of the clouds coming up or how scared the dogs are.

$\endgroup$
0
$\begingroup$

If the objective is to somehow combine the impacts of B and C on the probability of A, I think it make sense to evaluate the probabilities $P(A|B\cup C) $ and $P(A|B\cap C)$ where:

$P(A|B\cup C) =\frac{P(A|B)P(B)+P(A|C)P(C)-P(A\cap B\cap C)}{P(B\cup C)}$

and $P(A|B\cap C) =\frac{P(A\cap B \cap C)}{P(B \cap C)}$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.