5
$\begingroup$

I am trying to derive the normalizing constant for the multivariate Gaussian. The book I'm following suggests diagonalizing the covariance matrix and then using a change of variables.

So, we consider the following density for a random $d$-dimensional vector $\mathbf{x}$ and a positive definite symmetric matrix $\Sigma$. $$ p(\mathbf{x}) \propto e^{-\frac{1}{2}(\mathbf{x}-\mathbf{\mu})^T\Sigma^{-1}(\mathbf{x}-\mathbf{\mu})} $$

We can diagonalize $\Sigma=Q\Lambda Q^T$ and let $\mathbf{y}=Q(\mathbf{x}-\mathbf{\mu})$ and $z=\Lambda^{-1/2}\mathbf{y}$. Then $$ p(\mathbf{x}) \propto e^{-\frac{1}{2}\mathbf{z}^T\mathbf{z}} = e^{-\frac{1}{2}\sum_i \lambda_i y_i^2} $$ which is just a representation of the joint density of the independent $y_i$'s.

If everything's right, this should integrate to $$ \int_{-\infty}^{\infty}e^{-\frac{1}{2}\sum_i \lambda_i y^2_i}d\mathbf{y}=\sqrt{(2\pi)^d|\Sigma|} $$ which seems likely considering it the value of the Gaussian integral, though I'll admit that my calculus is still slightly beneath this one.

My questions are:

  • Does that last expression indeed integrate to that? Is it easier to integrate than the original density for $\mathbf{x}$, or was it all for nothing?
  • How can the change of variables formula be helpful here? This formula is $$ p_y(\mathbf{y}) = p_x(\mathbf{x}) \hspace{.2em} |\mbox{det } J_{x \rightarrow y}| $$ where $J_{x \rightarrow y}$ is the Jacobian matrix of $\mathbf{x}$ with respect to $\mathbf{y}$.
$\endgroup$
9
$\begingroup$
  • The reason that $\int_{\mathcal{R}^d} e^{-\frac{1}{2} \sum_i \lambda_i y_i^2} dy$ is easier to integrate is that it can be expressed as a product of univariate integrals:

\begin{align} \int_{\mathcal{R}^d} e^{-\frac{1}{2} \sum_i \lambda_i y_i^2} dy &= \int_{\mathcal{R}^d}\Pi_{i=1}^d e^{-\frac{1}{2} \lambda_i y_i^2} dy \\ &= \Pi_{i=1}^d \int_{-\infty}^{\infty} e^{\frac{-1}{2} \lambda_i y_i^2} dy_i \end{align}

Now we can apply the formula for integrating under a univariate normal distribution: \begin{align} \Pi_{i=1}^d \int_{-\infty}^{\infty} e^{\frac{-1}{2} \lambda_i y_i^2} dy_i &= \Pi_{i=1}^d \left(2 \pi \lambda_i \right) ^{-\frac{1}{2}} \\ &= \sqrt{ (2 \pi)^d \Pi_{i=1}^d \lambda_i } \end{align}

To finish this integral, note that the when you take the eigendecomposition $\Sigma = Q^T \Lambda Q$, the diagonal values of $\Lambda$ ($\lambda_i$) are the eigenvalues of $\Sigma$, and the product of the eigenvalues of $\Sigma$ is the determinant of $\Sigma$. That is, $\Pi_{i=1}^d \lambda_i = \mathrm{det}(\Sigma)$. This finally gives us $\int_{\mathcal{R}^d} e^{-\frac{1}{2} \sum_i \lambda_i y_i^2} dy = \sqrt{ (2 \pi)^d \left| \Sigma \right| }$.

  • But wait! How did we get the right answer already? Shouldn't there have been a Jacobian involved when we transformed from $x$ to $y$? We needed to prove that $\int_{\mathcal{R^d}} e^{-\frac{1}{2} (x - \mu)^T Q^T \Lambda^{-1} Q (x-\mu)} dx = \int_{\mathcal{R^d}} e^{-\frac{1}{2} y^T \Lambda^{-1} y} \left| \frac{\partial x}{\partial y} \right| dy = \sqrt{ (2 \pi)^d \left| \Sigma \right| }$, but we've only shown that $\int_{\mathcal{R^d}} e^{-\frac{1}{2} y^T \Lambda^{-1} y} dy = \sqrt{ (2 \pi)^d \left| \Sigma \right| }$.

To finish the proof, we need to show that $\left| J \right| = \left| \frac{\partial x}{\partial y} \right| = 1$. Let's look more carefully at that diagonalization.

Since $\Sigma$ is a covariance matrix, it should be symmetric positive definite. Therefore, there is an eigendecomposition where $Q$ is orthonormal, so $\Sigma = Q^{-1} \Lambda Q$, where $\Lambda$ is diagonal and $Q^{-1} = Q^T$.

So we have $y = Q(x - \mu)$, or $x = Q^T y + \mu$. Therefore, $\left| \frac{\partial x}{\partial y} \right| = |Q^T| = 1$, since $Q$ is orthonormal.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.