# Is the marginal distribution of a maximum entropy distribution also a maximum entropy distribution?

I am considering under what circumstances maximum entropy distributions are closed under marginalization. The main case I am interested is the following setting:

1. a finite set of discrete random variables $$X_{1},\ldots,X_{n}$$. To simplify notation, in the examples below I just use two random variables $$X,Y$$. The random variables are not independent.
2. a set of sufficient statistics $$T_{1}, \ldots, T_{m}$$ where $$T_{i}(x_{1},\ldots,x_{n})$$ returns a real number
3. a set of mean values $$\mu_{1},\ldots,\mu_{m}$$

I would like to start with the maximum entropy distribution $$P^{\ast}(X,Y)$$ that satisfies the moment matching constraints

$$E_{P}[T_{i}(X,Y)] = \mu_{i}.$$

It is well-known that this distribution can be represented in exponential form. Now consider the marginal sufficient statistics

$$T'_{i}(x) = \sum_{y} T_{i}(x,y).$$ with associated moment matching constraints $$E_{Q}[T'_{i}(X)] = \mu_{i}$$ and maximum entropy solution $$Q^{\ast}$$. Letting $$P^{\ast}(x) = \sum_{y} P^{\ast}(x,y)$$ be the marginal distribution for $$P^{\ast}$$, do we have that

$$P^{\ast}(x) = Q^{\ast}(x)?$$ If not true in general, are there conditions on the sufficient statistics that guarantee this condition?

Since $$Q^*$$ is of the form $$Q^*(x)\propto \exp\left\{\sum_j \lambda_j \mathbb{E}[T_j(x,Y)|X=x]\right\}$$ and $$P^*$$ is of the form $$P^*(x)\propto\sum_{y\in\mathcal Y} \exp\left\{\sum_j \zeta_j T_j(x,y)\right\}$$ and there is no reason this marginalisation property holds.
• Thank you for the reply. Actually $Q^{*}$ is the solution for the marginal sufficient statistics. So it is of the form $$Q^*(x)\propto \exp\left\{\sum_j \zeta_j T'_j(x)\right\}$$ Commented Mar 8, 2019 at 12:08