I am considering under what circumstances maximum entropy distributions are closed under marginalization. The main case I am interested is the following setting:

  1. a finite set of discrete random variables $X_{1},\ldots,X_{n}$. To simplify notation, in the examples below I just use two random variables $X,Y$. The random variables are not independent.
  2. a set of sufficient statistics $T_{1}, \ldots, T_{m}$ where $T_{i}(x_{1},\ldots,x_{n})$ returns a real number
  3. a set of mean values $\mu_{1},\ldots,\mu_{m}$

I would like to start with the maximum entropy distribution $P^{\ast}(X,Y)$ that satisfies the moment matching constraints

$$E_{P}[T_{i}(X,Y)] = \mu_{i}.$$

It is well-known that this distribution can be represented in exponential form. Now consider the marginal sufficient statistics

$$T'_{i}(x) = \sum_{y} T_{i}(x,y).$$ with associated moment matching constraints $E_{Q}[T'_{i}(X)] = \mu_{i}$ and maximum entropy solution $Q^{\ast}$. Letting $P^{\ast}(x) = \sum_{y} P^{\ast}(x,y)$ be the marginal distribution for $P^{\ast}$, do we have that

$$P^{\ast}(x) = Q^{\ast}(x)? $$ If not true in general, are there conditions on the sufficient statistics that guarantee this condition?


1 Answer 1


Since $Q^*$ is of the form $$Q^*(x)\propto \exp\left\{\sum_j \lambda_j \mathbb{E}[T_j(x,Y)|X=x]\right\}$$ and $P^*$ is of the form $$P^*(x)\propto\sum_{y\in\mathcal Y} \exp\left\{\sum_j \zeta_j T_j(x,y)\right\}$$ and there is no reason this marginalisation property holds.

  • $\begingroup$ Thank you for the reply. Actually $Q^{*}$ is the solution for the marginal sufficient statistics. So it is of the form $$Q^*(x)\propto \exp\left\{\sum_j \zeta_j T'_j(x)\right\}$$ $\endgroup$ Commented Mar 8, 2019 at 12:08
  • $\begingroup$ OK, I switched the notations then. $\endgroup$
    – Xi'an
    Commented Mar 8, 2019 at 12:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.