Marginalize Gaussian Random Variable completing the square [duplicate]

Question

I have $$P(x | \lambda_1, \lambda_2) = \mathcal{N}(\mu + L_1 \lambda_1 + L_2 \lambda_2, \Sigma)$$

and a Gaussian prior for $$P(\lambda_1) = P(\lambda_2) = \mathcal{N}(0, I)$$.

My task is to compute $$p(x | \lambda_1)$$ and I know the answer should be $$p(x | \lambda_1) = \mathcal{N}(\mu + L_1 \lambda_1, L_2 \Sigma L_2^T)$$.

I present below the steps I took and where I got stuck:

$\begin{align*} p(x | \lambda_1) &= \int p(x | \lambda_1, \lambda_2) p(\lambda_2) d\lambda_2 \\ &= \int const \cdot exp\big[(\hat{x} - L_2 \lambda_2)^T \Sigma^{-1} (\hat{x} - L_2 \lambda_2) + \lambda_2^T \lambda_2\big] d\lambda_2 \end{align*}$

Where $\hat{x} = x - \mu - L_1 \lambda_1$. I process the exponent and get the following: $\begin{align*} & (\hat{x} - L_2 \lambda_2)^T \Sigma^{-1} (\hat{x} - L_2 \lambda_2) + \lambda_2^T \lambda_2 \\ =& \hat{x}^T \Sigma^{-1} \hat{x} - 2 \hat{x}^T \Sigma^{-1} L_2 \lambda_2 + \lambda_2^T (L_2^T \Sigma^{-1} L_2 + I) \lambda_2 \end{align*}$

but I do not know how to proceed further. I have the notes for the solution for the exercise, but I do not understand it.

Below they also have $\lambda_1^T \lambda_1$ because the wrote $p(x | \lambda_1) = p(x | \lambda_1, \lambda_2) p(\lambda_1) p(\lambda_2)$ and then integrated $\lambda_2$ out, but I thought that's not necessary.

Help would be greatly appreciated.

Thank you!

Answer 1

This is essentially the problem of completing the square for a multivariate normal. Some of the steps in the notes are useful for you as well. Here I try to show the result by using the method you are trying.

\begin{align*} f(x \mid \lambda_1) &= \int f(x \mid \lambda_1, \lambda_2) f(\lambda_2)\, d\lambda_2\\ & \propto \int \exp \left\{-\dfrac{1}{2}\left((x - \underbrace{\mu + L_1 \lambda_1}_{a} + L_2 \lambda_2)^T \Sigma^{-1}(x - \mu + L_1 \lambda_1 + L_2 \lambda_2) + \lambda^T_2\lambda_2 \right) \right\}\\ & = \int \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) -2(x - a)^T\Sigma^{-1}L_2\lambda_2 +\lambda_2^T(L_2^T\Sigma^{-1}L_2)\lambda_2 + \lambda^T_2\lambda_2 \right) \right\}\\ & = \int \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) -2(x - a)^T\Sigma^{-1}L_2\lambda_2 + \lambda_2^T\underbrace{(L_2^T\Sigma^{-1}L_2 +I)}_{B}\lambda_2 \right) \right\}\,. \end{align*}

At this point note that $B$ is symmetric and invertible and we artifically introduce $B^{-1}B$ in order to complete the square.

\begin{align*} f(x \mid \lambda_1) &\propto \int \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) -2(x - a)^T\Sigma^{-1}L_2B^{-1}B\lambda_2 + \lambda_2^TB\lambda_2 \right) \right\}\\ & = \exp \left\{-\dfrac{1}{2}(x - a)^T\Sigma^{-1}(x-a) \right\} \int \exp \left\{-\dfrac{1}{2}\left( -2\left(B^{-1}L_2^T\Sigma^{-1}(x - a)\right)^TB\lambda_2 + \lambda_2^TB\lambda_2 \right) \right\} \end{align*} Note that inside the integral the term can be made into the kernel of a multivariate normal with mean $\left(B^{-1}L_2^T\Sigma^{-1}(x - a)\right)^T$.So we add and subtract the square of this mean (not that this term is not dependent on $\lambda_2$. \begin{align*} &f(x \mid \lambda_1)\\ & \propto \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) \right) \right\} \int \exp \Big\{-\dfrac{1}{2} \Big( -2\left(B^{-1}L_2^T\Sigma^{-1}(x - a)\right)^TB\lambda_2 + \lambda_2^TB\lambda_2 \\ & + (B^{-1}L_2^T\Sigma^{-1}(x-a))^TB (B^{-1}L_2^T\Sigma^{-1}(x-a)) - (B^{-1}L_2^T\Sigma^{-1}(x-a))^TB (B^{-1}L_2^T\Sigma^{-1}(x-a)) \Big) \Big\}\\ & = \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) - (B^{-1}L_2^T\Sigma^{-1}(x-a))^TB (B^{-1}L_2^T\Sigma^{-1}(x-a))\right) \right\} \\ & \times \int \exp \Big\{-\dfrac{1}{2} \Big( \lambda_2^TB\lambda_2 -2\left(B^{-1}L_2^T\Sigma^{-1}(x - a)\right)^TB\lambda_2\\ & \quad + (B^{-1}L_2^T\Sigma^{-1}(x-a))^TB (B^{-1}L_2^T\Sigma^{-1}(x-a)) \Big) \Big \}\\ & = \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) - (B^{-1}L_2^T\Sigma^{-1}(x-a))^TB (B^{-1}L_2^T\Sigma^{-1}(x-a))\right) \right\} \\ & \times \int \exp \Big\{-\dfrac{1}{2} \Big( \left(\lambda_2 - \left(B^{-1}L_2^T\Sigma^{-1}(x - a)\right) \right)^TB\left(\lambda_2 - \left(B^{-1}L_2^T\Sigma^{-1}(x - a)\right) \right) \Big) \Big \}\\ & \propto \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) - (B^{-1}L_2^T\Sigma^{-1}(x-a))^TB (B^{-1}L_2^T\Sigma^{-1}(x-a))\right) \right\}\,. \end{align*}

So now, you have integrated out $\lambda_2$ and what is left is to complete the square similarly for $x$, and ignore all terms that are not dependent on $x$.

\begin{align*} &f(x \mid \lambda_1)\\ &\propto \exp \left\{-\dfrac{1}{2}\left((x - a)^T\Sigma^{-1}(x-a) - (B^{-1}L_2^T\Sigma^{-1}x)^TB (B^{-1}L_2^T\Sigma^{-1}x)\right) \right\}\\ & \propto \exp \left\{-\dfrac{1}{2}\left(x^T \Sigma^{-1}x - 2a^T\Sigma^{-1}x - x^T\Sigma^{-1}L_2 B^{-1}L_2^T\Sigma^{-1}x \right) \right\}\\ & = \exp \left\{-\dfrac{1}{2}\left(x^T \left(\Sigma^{-1} - \Sigma^{-1}L_2 B^{-1}L_2^T\Sigma^{-1} \right)x - 2a^T\Sigma^{-1}x \right) \right\}\\ & = \exp \left\{-\dfrac{1}{2}\left(x^T \left(\Sigma^{-1} - \Sigma^{-1}L_2 B^{-1}L_2^T\Sigma^{-1} \right)x - 2a^T\Sigma^{-1}\left(\Sigma^{-1} - \Sigma^{-1}L_2 B^{-1}L_2^T\Sigma^{-1} \right)^{-1} \left(\Sigma^{-1} - \Sigma^{-1}L_2 B^{-1}L_2^T\Sigma^{-1} \right) x \right) \right\} \end{align*}

Which gives $x \mid \lambda_1$ is a Normal distribution with mean $a(I - L_2BL_2^{-1} \Sigma^{-1})^{-1})$ and covariance $(\Sigma^{-1} - \Sigma^{-1}L_2B^{-1}L_2^T\Sigma^{-1})^{-1}$. I don't quite get the answer you said you expect.

Answer 2

Managed to figure out what's going on.

We have $\lambda_2^T B \lambda_2$ which is the square term and we now need $-2 Q^T B \lambda_2$ where Q is some term we need to determine.

Since we don't have $B$ over there, $Q$ must contain $B^{-1}$. Important to notice that $B$ is symmetric, so $B^{-1}$ is symmetric and hence $(B^{-1})^T = B^{-1}$.

The observations above give us $Q = B^{-1} L_2^T \Sigma^{-1} \hat{x}$.

Now that we've got the $-2 Q^T B \lambda_2$ term, we need $Q^T B Q$ to complete the square $(\lambda_2 - Q)^T B (\lambda_2 - Q)$. We add and subtract $Q^T B Q$ to do this.

The last step just involves putting everything within $\hat{x}^T H \hat{x}$ and marginalizing the $\lambda_2$ Gaussian (easy to do since the integral evaluates to 1).

Hope I didn't make any mistakes.