Intuition about the relation between joint distribution, marginal distribution, and conditional distribution

The wording "intuition" might be a bit imprecise. I want to discuss how we visualize in our head going from one to another among the joint PDF, marginal PDF, and conditional PDF.

To make the discussion simple, let's assume the joint PDF $$f(x,y)$$ is defined on some bounded set in $$\mathbb{R}^2$$, like $$[0,1]\times[0,1]$$.

Imagine the joint PDF $$f(x,y)$$ is plotted on the unit square $$[0,1]\times[0,1]$$.

The intuition behind getting the marginal distribution from the joint distribution is simple. We just "collapse" the joint PDF along, say, the x-axis by integrating $$f(x,y)$$ over $$x$$ to get $$f(y)$$. The integration of $$f$$ along $$x$$ is like swatting a mosquito--what was three-dimensional has become two-dimensional (strictly speaking, in this case of the PDF of two variables, it's some two-dimensional surface that has become a one-dimensional curve.)

The intuition behind getting the conditional distribution is also simple. If we want to know $$f(x|y_0)$$, the PDF of $$x$$ given certain $$y_0$$, we just cut $$f(x,y)$$ along the line $$y=y_0$$ and look at the shape of the cross-section. That's all that matters--the shape. The real conditional distribution is just this shape but vertically stretched or compressed (i.e. normalized). How much it is stretched or compressed depends on the value of $$y_0$$ chosen.

If we go the other way around, we're not guaranteed to get a unique joint PDF from a pair of known marginal PDF's, or a pair of known conditional PDF's. Of course, we assume the dependence between $$x$$ and $$y$$ here.

The reason that this is not guaranteed by a pair of known marginal PDF's is somewhat straightforward. Imagine we've got a candidate for the joint PDF. Inspired by the result of $$x$$ and $$y$$ being independent, we may choose $$f(x)f(y)$$ as our candidate. Indeed, "swatting" this candidate in the direction of, say, $$x$$ gives the desired $$f(y)$$. But we can mask this candidate arbitrarily with something that disappears when you swat at it in either direction--something that integrates to 0 w.r.t. $$x$$ or $$y$$. Just add this disappearing crumpled sheet to our candidate surface and we get another valid candidate. So, it's not unique. Another way of thinking is that some information about the two-dimensional surface has been lost if we only know what it looks like respectively after being smashed onto two different walls.

But what about going from the conditionals to the joint? What bothers me is that, knowing the conditional PDF's of $$x$$ given every possible $$y$$ and the conditional PDF's of $$y$$ given every possible $$x$$ seems to suggest that we know the "shapes" of the cross-section of the joint PDF dissected at every value of $$y$$ and at every value of $$x$$, and that seems to be a strong enough statement to get the full shape of $$f(x,y)$$. The whole ideas of those whimsical "intuitions" seem to have fallen apart.

I'm aware that what I've said could have some serious errors that'd make the whole "intuition" gibberish even more non-sensical. In that case please point them out. In fact, anything will be appreciated.

-------some bonus nonsense below---------

Extending the math behind the bold font question, it seems that we can write it as:

Given the conditional PDF's $$k_x:[0,1]\times[0,1]\rightarrow \mathbb{R}^+$$ (since the conditional PDF is known for every possible $$y$$, this function maps from a subset of $$\mathbb{R}^2$$) and $$k_y:[0,1]\times[0,1]\rightarrow \mathbb{R}^+$$ (same here), s.t. $$\int_0^1k_x(x,y)\text{d}x = 1$$ and $$\int_0^1k_y(x,y)\text{d}y = 1$$, we want to find the marginals $$f_x:[0,1]\rightarrow\mathbb{R}^+$$ and $$f_y:[0,1]\rightarrow\mathbb{R}^+$$ s.t. \begin{align}\int_0^1f_x(x)\text{d}x&=1,\\\int_0^1f_y(y)\text{d}y&=1,\\ \text{and } k_x(x,y)f_y(y)&=k_y(x,y)f_x(x).\end{align}

Is $$f_x(x)$$ and $$f_y(y)$$ unique? If not, what are the examples?

Also, what seems a bit bizarre to me is that, $$k_x(x,y)f_y(y)=k_y(x,y)f_x(x)$$ suggests we can write $$\frac{k_x(x,y)}{k_y(x,y)}$$ as the product between a function of $$x$$ and a function of $$y$$. However, assuming $$x$$ and $$y$$ are mutually dependent, neither $$k_x(x,y)$$ nor $$k_y(x,y)$$ can be individually written as such a product, because otherwise the integration of, say, $$k_x$$ over $$x$$ is not the constant 1, but a function of $$y$$. Is this true? What does this tell us about the possible set of conditional PDF's $$k_x$$ and $$k_y$$? What could be the examples?

• In Gibbs sampling, you can simulate draws from the joint distribution of x and y by alternating draws from the two conditionals. So, I am inclined to say that specifying the two conditionals is enough to uniquely pin down the joint distribution. You could check the proof of convergence of the Gibbs sampler and see if what you need is there. Jun 18, 2022 at 23:24
• "specifying the two conditionals is enough to uniquely pin down the joint distribution" --- you may need some conditions there; Gibbs sampling doesn't always work. Jun 20, 2022 at 1:15
• @Glen_b I've posted another question specifically addressing finding the joint from the conditionals. stats.stackexchange.com/questions/579269 I'd really appreciate it if you could shed some light on that! Jun 20, 2022 at 2:17
• I think there's some people much better placed than me to answer that question, but I'll see how time goes. Jun 20, 2022 at 2:20
• You can find a proof here for the discrete case: <les-mathematiques.net/vanilla/index.php?p=discussion/comment/…>. Jul 13, 2022 at 12:26