This is a sort of paradox that has me confused. i just need some help clearing things up.

Let $x,y$ be two discrete, finite variables, where $x$ can take on $N$ values and $y$ can take $M$ values. Suppose someone hands you a table of the numbers $P(x|y)$. Can you determine $P(x,y)$? Intuitively, I think the answer is No. However, consider the following system of equations:

$$P(x|y)\sum_\xi P(\xi,y) = P(x,y)$$

(This is just another way to write the product rule $P(x|y)P(y)=P(x,y)$.)

Here you have $M\times N$ linear equations in the $M\times N$ unknowns $P(x,y)$. So it seems that $P(x,y)$ could be determined from this system of equations. If this is true, then knowledge of the $M\times N$ numbers $P(x|y)$ are enough to determine the $M\times N$ numbers $P(x,y)$, but this doesn't feel right. What am I doing wrong?

  • $\begingroup$ You have MxN + M unknowns since you don't know P(y) (without considering constraints like all P(y) sum to 1). $\endgroup$ – George Jan 17 '16 at 13:53
  • $\begingroup$ @George $P(y) = \sum_\xi P(\xi, y)$. $\endgroup$ – becko Jan 17 '16 at 13:54
  • $\begingroup$ @George I realized my mistake by reading your comment. I don't know if this is what you had in mind, but I've posted an answer. Thanks! $\endgroup$ – becko Jan 17 '16 at 14:22
  • $\begingroup$ Why the minus vote? Please leave a comment. $\endgroup$ – becko Jan 19 '16 at 2:40

Your $\sum_\xi P(\xi,y)$ is just a fancy way of writing $P(y)$, so your formula for $P(x,y)$ is the product of $P(x|y)$ and $P(y)$. And so indeed $P(x,y)$ was not determined just by $P(x|y)$.

  • $\begingroup$ No, the system of equations I wrote only involves the unknowns $P(x,y)$, and the known values $P(x|y)$. $\endgroup$ – becko Jan 17 '16 at 13:57
  • $\begingroup$ Your $P(\xi,y)$ are in the equations. That's not "Kosher" per your theory. Given that what you propose is wrong, you need to open your mind a bit to see your error. $\endgroup$ – Mark L. Stone Jan 17 '16 at 14:01
  • $\begingroup$ I agree there is an error, I just don't agree that it is what you point out. If I understand correctly, what you're saying is that in my system of equations, $P(x,y)$ is determined not only by $P(x|y)$, but by both $P(x|y)$ and $P(y)$. However, I've purposefully written the equations in such a way that $P(y)$ does not appear explicitly. What I have written is a system of equations in the unknowns $P(x,y)$, where only $P(x|y)$ appears. $\endgroup$ – becko Jan 17 '16 at 14:14

The numbers $P(x|y)$ are not $M\times N$ independent numbers, because of the $N$ constrains $\sum_x P(x|y) = 1$ (one for each value of $y$). This means that in the systems of equations above you only have $M\times N - N$ independent constrains.

However, $P(x,y)$ are $M\times N - 1$ independent numbers (the $-1$ comes from the constrain $\sum_{x,y} P(x,y) = 1$). Hence you need to specify an additional $N-1$ numbers, which usually come from specifying $P(y)$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.