If marginal probabilities equal, can we say anything about joint distribution? Consider I have equal marginal probabilities as per:
$$p_{\theta}(x) = p_{\alpha}(x).$$
Now assume we know there exists another random variable $z$ that is of interest to us. Am  I correct in thinking that I can not deduce that:
$$p_{\theta}(x, z) = p_{\alpha}(x, z)?$$
I believe this would be the case because you can get the same marginal from different joint distributions, but I am unsure if that applies here.
 A: You are correct. For example, suppose we have two bivariate normals. In the first, both $x$ and $z$ have mean 0 and variance 1, and correlation 0 (i.e. they are independent). In the second, both $x$ and $z$ still have mean 0 and variance 1, but their correlation is 0.5 (they are not independent). In both cases, the marginal distribution of $x$ will be normal with mean 0 and variance 1, but the joint distribution of $(x,z)$ will be different due to the different correlation values.
A: The restrictions on marginals implies some restriction on the joint, namely
$$\sum_z p_\theta(x, z)  = \sum_z p_\alpha(x, z)$$
This is exactly one degrees of freedom, meaning that if you fix the all $n$ components of $p_\theta(x, z) $, then you are at liberty to choose $n-1$ element for $p_\alpha(x, z) $ anyway you want, with the last degree of freedom fixed by
$$p_\alpha(x, z_n) = \sum_z p_\theta(x, z) - \sum_{z\neq z_n} p_\alpha(x, z)$$.
This might be a small constraint or a relatively large one when the support in the $z$ dimension is a small set, think n=3 or n=2!
