Prove that the maximum entropy distribution with a fixed covariance matrix is a Gaussian I'm trying to get my head around the following proof that the Gaussian has maximum entropy.
How does the starred step make sense? A specific covariance only fixes the second moment. What happens to the third, fourth, fifth moments etc?

 A: The starred step is valid because (a) $p$ and $q$ have the same zeroth and second moments and (b) $\log(p)$ is a polynomial function of the components of $\mathbf{x}$ whose terms have total degrees $0$ or $2$.

You need to know only two things about a multivariate normal distribution with zero mean:


*

*$\log(p)$ is a quadratic function of $\mathbf{x}=(x_1,x_2,\ldots,x_n)$ with no linear terms.  Specifically, there are constants $C$ and $p_{ij}$ for which $$\log(p(\mathbf{x}))=C + \sum_{i,j=1}^n p_{ij}\, x_i x_j.$$
(Of course $C$ and the $p_{ij}$ can be written in terms of $\Sigma$, but this detail does not matter.)

*$\Sigma$ gives the second moments of the distribution.  That is, $$\Sigma_{ij}=E_p(x_i x_j) = \int p(\mathbf{x})\, x_ix_j\, d\mathbf{x}.$$
We may use this information to work out an integral:
$$\eqalign{
& &\int(q(\mathbf{x}) - p(\mathbf{x}))\log(p(\mathbf{x}))d\mathbf{x}  \\
&= &\int(q(\mathbf{x}) - p(\mathbf{x}))\left(C + \sum_{i,j=1}^n p_{ij}\, x_i x_j\right)d\mathbf{x}.
}$$
It breaks into the sum of two parts:


*

*$\int(q(x) - p(x))C\, d\mathbf{x} = C\left(\int q(\mathbf{x}) d\mathbf{x} - \int p(\mathbf{x}) d\mathbf{x}\right) = C(1 - 1) = 0$, because both $q$ and $p$ are probability density functions.

*$\int(q(\mathbf{x}) - p(\mathbf{x})) \sum_{i,j=1}^n p_{ij}\, x_i x_jd\mathbf{x} = \sum_{i,j=1}^n p_{ij}\int(q(\mathbf{x}) - p(\mathbf{x}))x_i x_jd\mathbf{x} = 0$ because each of the integrals on the right hand side, $\int q(\mathbf{x}) x_i x_jd\mathbf{x}$ and $\int p(\mathbf{x}) x_i x_jd\mathbf{x}$, has the same value (to wit, $\Sigma_{ij}$).  This is what the remark "yield the same moments of the quadratic form" is intended to say.
The result follows immediately: since $\int(q(\mathbf{x}) - p(\mathbf{x}))\log(p(\mathbf{x}))d\mathbf{x}=0$, we conclude that $\int q(\mathbf{x})\log(p(\mathbf{x}))d\mathbf{x} = \int p(\mathbf{x})\log(p(\mathbf{x}))d\mathbf{x}.$
A: I think what happens is that in the integrals in both (4.27) and (4.28) you have $q(x)$ and $p(x)$ multiplying terms of the form $\sigma_{ij}x_ix_j$ (because $p(x)$ is a  normal density, when you take the log you obtain just such kind of terms from the exponent plus constants). But then the condition in the theorem ensures that those terms multiplied by either $p(x)$ of $q(x)$ integrate to the same value.
A: We can also verify the above result by solving the constrained optimization problem. For simplicity, let's prove for one dimensional random variable $X$ first (with $\Sigma=[\sigma^2]$ fixed, with $V(X) = E[(X-\mu)^2]=\sigma^2$, where $E[X]=\mu$), that can be generalized to multliple dimensions (random vectors). Let the pdf of $X$ be $p(.)$, then the constrained maximization problem becomes the following:
$\begin{align}
\underset{p}{maximize \;} H(p) & \\
s.t. \int{(x-\mu)^2p(x)dx} &=\sigma^2 \; \text{and}\\
\int{p(x)}dx=1
\end{align}$,
where entropy $H(p)=-\int{p(x)\log p(x)}dx$ and the integrals are evaluated over the range of $X$.
The Lagrangian
$\mathcal{L}(p(x), \lambda_1, \lambda_2)=-\int{p(x)\log p(x)}dx + \lambda_1\left( \int{(x-\mu)^2p(x)dx} - \sigma^2\right)+\lambda_2\left(\int{p(x)}dx-1\right)$
$\therefore \frac{\partial \mathcal{L}}{\partial p(x)}$
$=\int{\left(-(1+\log p(x))+\lambda_1(x-\mu)^2+\lambda_2\right)}dx = 0$, at a critical point
$\Rightarrow \log p(x) = \lambda_1(x-\mu)^2+\lambda_2-1$
$\Rightarrow p(x) = e^{-\lambda_1(x-\mu)^2+\lambda_2-1}$
$\Rightarrow p(x) \propto e^{-\lambda_1(x-\mu)^2}$, which is a Gaussian p.d,f.
