Consider a set of random variables $\{X_i\}$ with joint pdf $f(x_1 ... x_n)$. Given the marginal pdfs $f_i(x_i)$, we can construct a joint distribution $$g(x_1 ... x_n) = \prod_i f_i(x_i)$$ which has the same marginals as $f$. (In particular, $g$ is the maximum entropy distribution satisfying this constraint.) $g$ is in some sense an approximation to $f$.

If we know the pairwise joint marginals $f_{ij}(x_i,x_j)$, is it possible to do something similar? Specifically:

  1. What conditions do the $f_{ij}$ need to satisfy in order for there to exist a $g$ giving pairwise joint marginals $f_{ij}$?
  2. Is there an explicit formula or simple algorithm that gives any such $g$?
  3. Is there an explicit formula that gives the maximum-entropy such $g$?

I'm also interested in generalizations to $k$-wise joint marginals, and the conditions under which the corresponding sequence of approximations converges to $f$, but that seems like a separate question. An answer for the special case where all the $f_{ij}$ are the same would also be interesting.

See also:
Constructing a joint distribution from pairwise bivariate marginal distributions? (No answers, but a comment mentions "pair-copula model")

  • $\begingroup$ The pairwise marginal densities need be compatible with at least one joint$$f_{ij}(x_i,x_j)=\int f(\mathbf x)\text d\mathbf x_{-(i,j)}$$ $\endgroup$ – Xi'an Jan 8 at 8:26
  • $\begingroup$ @Xi'an right, of course, I guess really I'm asking for a more explicit condition (efficiently computable, maybe). $\endgroup$ – Daniel Jan 9 at 5:43
  • 1
    $\begingroup$ I don't have a direct answer to your question, but the "pair-copula" construction mentioned in the other question is probably not what you want (e.g. for 3 variables, it constructs the joint from 3 pairs, but it's not $(X_1,X_2)$, $(X_2,X_3)$ and $(X_1,X_3)$, it's $(X_1,X_2)$, $(X_2,X_3)$, and $(X_1,X_3|X_2)$). $\endgroup$ – Chris Haug Jun 12 at 14:49

For discrete random variables the raking/iterative proportional fitting algorithm constructs a joint distribution if one exists (under some additional assumption about zero cells). It works for marginal distributions of any order, and not necessarily the same order for each margin.

IPF is (or was) used to fit loglinear models -- its guaranteed convergence rate is only linear, but it use less memory than Newton-type methods. Raking is used in survey statistics to match sample margins to known population margins.

I don't know if this generalises to continuous distributions.

  • $\begingroup$ The algorithm described in the Wikipedia article you link looks like it only enforces that the single-variable marginals match a target distribution. Is there some way of generalizing it to enforce a constraint on the two-variable joint marginals? $\endgroup$ – Daniel Jan 8 at 4:01
  • $\begingroup$ Yes. It's the same algorithm (classic IPF): you reweight the observations so one pairwise marginal distribution is correct and then iterate over all the pairwise margins. It wouldn't be much use if it only worked for univariate margins. There's an example here: stats.stackexchange.com/questions/37771/… $\endgroup$ – Thomas Lumley Jan 8 at 4:21
  • $\begingroup$ Ok, this makes sense, and probably one can do something vaguely similar with continuous distributions using a sequence of finitely-parameterized spaces. Unfortunately, I don't think this algorithm provides much insight into the structure of the resulting distribution, since it's nearly a black-box optimization algorithm. $\endgroup$ – Daniel Jan 14 at 18:19
  • 1
    $\begingroup$ In the discrete case, the resulting distribution is the log-linear model whose sufficient statistics are the supplied margins. $\endgroup$ – Thomas Lumley Jan 27 at 2:19
  • $\begingroup$ Ok, that's interesting, thanks! $\endgroup$ – Daniel Jan 30 at 0:10

The general framework is to select from the set of all joint distributions matching a set of statistics the unique distribution with maximum entropy.

For pair-wise covariance statistics, see:
Dempster, Arthur P. Covariance selection. Biometrics (1972): 157-175.

For arbitrary "filters" (functions of many variables), see:
Zhu, S. C., Y. Wu, and D. Mumford. FRAME: Filters, Random Fields and Maximum Entropy - Towards a Unified Theory for Texture Modeling. IEEE CVPR 1993.

(Note similarities in approach. Dempster, Mumford, Wu and Zhu were all at Harvard in the early 1990s.)


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.