8
$\begingroup$

I am working my way (self-study) through E.T. Jaynes' book Probability Theory - The Logic of Science

Original Problem

Exercise 2.1 says: "Is it possible to find a general formula for $p(C|A+B)$ analogous to [the formula $p(A+B|C)=p(A|C)+p(B|C)-p(AB|C)$] from the product and sum rules. If so, derive it; if not, explain why this cannot be done."

Givens

The rules I have to work with are:

$p(AB | C) = p(A|C)p(B|AC) = p(B|C)p(A|BC)$ and $p(A|B)+p(\bar{A}|B)=1$

Where we can also use logical identities to manipulate propositions. For example: $A+B=\overline{\bar{A}\bar{B}}$

Assumption of Solvability

I believe it must be possible because he does not introduce any other rules later and having a simple logical combination of propositions that was not easily expressible would defeat Jaynes' central thesis. However, I've been unable to derive the rule.

My Attempt

To keep myself from getting confused due to using the same variable names as the givens, I am solving the problem as:

Derive a formula for $p(X|Y+Z)$

Introducing a tautology for conditioning

My best attempt at solving it so far has been to introduce a proposition $W$ which is always true. Thus I can rewrite $Y+Z$ as $(Y+Z)W$ (since truth is the multiplicative identity).

Then, I can write:

$p(X|Y+Z)=p(X|(Y+Z)W)$

So, rewriting one of the givens as Bayes' rule: $p(A|BC)=\frac{p(B|AC)p(A|C)}{p(B|C)}$, I can write:

$p(X|(Y+Z)W)=\frac{p(Y+Z|XW)p(X|W)}{p(Y+Z|W)}=\frac{p(Y+Z|X)p(X|W)}{p(Y+Z|W)}$

Why this doesn't work

The term $p(Y+Z|X)$ is easy to deal with. (Its expansion is referred to in the problem definition.)

However, I don't know what to do with $p(X|W)$ and $p(Y+Z|W)$. There is no logical transformation I can apply to get rid of the $W$, nor can I think of any way of applying the given rules to get there.

Other places I've looked

I've done a Google search, which turned up this forum page. But the author does the same thing I tried without seeing the difficulty I have with the resulting conditioning on the introduced tautology.

I also searched stats.stackexchange.com for "Jaynes" and also for "Exercise 2.1" without finding any useful results.

$\endgroup$
  • $\begingroup$ I don't think this is worthy of its own answer, but I think what you have is correct, and is what Jaynes was expecting you to come up with. You can assume $W$ to be True always. And the next question he gives is about the more general case of $p(C|(A_1 + A_2 + ... + A_n)W)$, in which he asks you to prove the more general form of Bayes Theorem. (This is years after this question was posed, but I was stuck on the same part. I hope this comment helps others) $\endgroup$ – William Oliver Feb 7 '17 at 3:55
3
$\begingroup$

I am not sure what Jaynes considers to be analogous to $P(A\cup B \mid C) = P(A \mid C) + P(B \mid C) - P(AB \mid C)$ but students have cheerfully used one or more of the following on homework and exams: $$ \begin{align*} P(A\mid B \cup C) &= P(A \mid B) + P(C)\\ P(A\mid B \cup C) &= P(A \mid B) + P(C) - P(AC)\\ P(A\mid B \cup C) &= P(A \mid B) + P(A \mid C),\\ P(A\mid B \cup C) &= P(A \mid B) + P(A \mid C) - P(A \mid BC),\\ P(A\mid B \cup C) &= P(AB \mid B\cup C) + P(AC \mid B \cup C) - P(ABC \mid B\cup C). \end{align*} $$ Do you think any of these are correct?

Note: Changing my (now-deleted) comment into an addendum to my answer, the rules permit the following manipulations: $P(AB \mid C) = P(A\mid C)P(B \mid AC); P(A \mid C) = 1 - P(A^c \mid C).$ The first introduces conditioning on a subset of $C$ but does not eliminate conditioning on $C$. The second also does not eliminate conditioning on $C$. So any manipulations of $P(A\mid B \cup C)$ will always include terms of the form $P(X\mid B \cup C)$, and $P(A\mid B \cup C)$ cannot be expressed in terms of $P(A \mid B)$, $P(A \mid C)$, $P(A \mid BC)$, etc. without including probabilities conditioned on $B \cup C$ also.

$\endgroup$
  • $\begingroup$ I don't like any of them. I think the last answer is technically correct, but it doesn't eliminate the conditioning I am trying to remove because all of the terms still have a function of $A$ given $B \cup C$. $\endgroup$ – Eponymous Oct 10 '11 at 21:44
  • $\begingroup$ @Eponymous But notice that in the expression that Jaynes wants you to emulate, viz. $P(A\cup B\mid C) =P(A\mid C) + P(B\mid C) − P(AB\mid C)$, the conditioning remains on $C$ throughout. So the question boils down to what Jaynes thinks as analogous. My last "identity" is a true statement, and has conditioning on the same event on both sides, just like Jaynes's $P(A\cup B\mid C) =P(A\mid C) + P(B\mid C) − P(AB\mid C)$ does $\endgroup$ – Dilip Sarwate Oct 10 '11 at 22:03
  • $\begingroup$ In Jaynes' framework you must always condition on something. $P(X)$ is undefined for all $X$. Thus conditioning on a single term is unavoidable. My understanding of the problem is that I need to somehow break $P(A|B \cup C)$ into things that might be more natural/easier to calculate, ideally removing the $\cup$ altogether (though it may be irremovable). I don't see your last identity as doing that. $\endgroup$ – Eponymous Oct 11 '11 at 12:10
  • $\begingroup$ @Dilip I don't think there's anything wrong with your answer, and I think there's a lot (especially the last bit) which are quite right (+1). Absent a solution manual (which I don't think even exists), we can never know what was on Jaynes' mind. But I think you're right that the final answer should have conditioning all the way through, and what would seem to be the natural conditioning event would be: $A \cup B$. $\endgroup$ – user1108 Nov 10 '11 at 23:15
2
$\begingroup$

For problems like this one, it is sometimes helpful to think less about the formulas and instead draw a picture (in this case, a Venn diagram).

enter image description here

Now stare at the picture and try to visualize what $P(C | A \cup B)$ represents. If you can pick it out of the picture, then you will see that there are several valid ways to write it (two ways jump to my mind off the bat). If you're still stuck, try going back to the usual proof of the ordinary general addition rule for hints.

Remember: a conditional probability concentrates all of its probability mass on the conditioning event (in this case, $A \cup B$). The idea is to focus on the locations where $C$ intersects that event.

By the way, the R code for the figure is

library(venneuler)
vd <- venneuler(c(A=0.2, B=0.2, C=0.2, "A&B"=0.04, "A&C"=0.04, "B&C"=0.04 ,"A&B&C"=0.008))
plot(vd)
$\endgroup$
  • $\begingroup$ How does this help in writing a formula using Jaynes's rules only? We are supposed to use only the two rules stated by the OP. $\endgroup$ – Dilip Sarwate Nov 10 '11 at 20:44
  • 1
    $\begingroup$ @Dilip I think the hardest part of this Jaynes problem is that he didn't explicitly state the formula to shoot for. But the diagram allows us to see potential formulas which have a chance in Heck to be valid, and yes, the one I'm thinking of can be proved with just the product and sum rules (in fact, Jaynes did it in the paragraph immediately proceeding the original exercise!). $\endgroup$ – user1108 Nov 10 '11 at 21:23
  • $\begingroup$ @Jay The problem, as I pointed out in the comments on my own answer, is that everything in a formula for $P(C \mid A\cup B)$ necessarily needs to be conditioned on $A\cup B$. On the other hand, as you say, Jaynes's text proves the conditional version of the well-known result: $P(A \cup B) = P(A) + P(B) - P(AB)$. It is easy to break apart the conditioned event, not so for the conditioning event. $\endgroup$ – Dilip Sarwate Nov 10 '11 at 23:42
  • $\begingroup$ @Dilip yep, can't break up the conditioning event, I'm with you. $\endgroup$ – user1108 Nov 11 '11 at 0:02
1
$\begingroup$

Bayes Theorem gives $$ p(C\mid A+B) = \frac{p(A+B\mid C)\;p(C)}{p(A+B)} \, . $$ Now, using the conditional and unconditional sum rules, we have $$ p(C\mid A+B) = \frac{p(A\mid C)+p(B\mid C)-p(AB\mid C)}{p(A)+p(B)-p(AB)}\;p(C) \, . $$ Of course, the question is whether or not this formula would be "analogous enough" for Jaynes.

$\endgroup$
  • $\begingroup$ As the OP pointed out in a comment, "In Jaynes' framework you must always condition on something. P(X) is undefined for all X. Thus conditioning on a single term is unavoidable." So, you are not allowed to write P(C), P(A), etc. $\endgroup$ – Dilip Sarwate Feb 26 '12 at 0:03
1
$\begingroup$

You can't get rid of the tautology. I think you are supposed to just add the tautology and apply the product rule and then the sum rule and you get:

$$p(C|(A+B)W) = \frac{p(CA|W)+p(CB|W)-p(AB|W)}{p(A|W)+p(B|W)-p(AB|W)}$$

where all the probabilities are expressed as posteriors to the tautology. I think this is the most similar equivalent to the sum rule that you can get for this problem, so that would be the solution.

Note that if you add the condition $p(AB|W)=0$ (i.e. $A$ and $B$ are mutually exclusive) you get the same expression that you have to prove in the problem 2.2, that would indicate this solution is most probably correct (by Bayesian induction ;).

$\endgroup$
0
$\begingroup$

Following only the Cox's rules, taking $W=X$ as in Jaynes's book, we have the solution from MastermindX:

$$ p(C|(A+B)X) = \dfrac{p(C(A+B)|X)}{p((A+B)|X)} \qquad \text{(product rule)}$$ $$ = \dfrac{p((CA+CB)|X)}{p((A+B)|X)} \qquad \text{(distributive property of the conjunction)}$$ $$ = \dfrac{p(CA|X)+p(CB|X)-p(CAB|X)}{p((A+B)|X)} \qquad \text{(sum rule on numerator)}$$ $$ = \dfrac{p(CA|X)+p(CB|X)-p(CAB|X)}{p(A|X)+p(B|X)-p(AB|X)} \qquad \text{(sum rule on demoninator)}$$ $$ = \dfrac{p(A|X)p(C|AX)+p(B|X)p(C|BX)-p(AB|X)p(C|ABX)}{p(A|X)+p(B|X)-p(AB|X)} \qquad \text{(product rule on numerator)}$$

The solution for Ex. 2.1 follows the intention of the Chapter 2 in the product rule, that "we first seek a consistent rule relating the plausibility of the logical product $AB$ to the plausibility of $A$ and $B$ separately" (page 24). Moreover, for mutually exclusive propositions $A$ and $B$, this is equal to the Eq. (2.67) in Ex. 2.2, if we take $\{A_{1} = A$, $A_{2} = B\}$; also indicated by MastermindX. Notice that Jaynes himself does not get rid of the additional information $X$ on Eq. (2.67), so I believe this is the expected solution for both exercises.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.