Computing the Gaussian posterior from likelihood and prior Say I have a gaussian likelihood and prior,
$$
p(\theta) = \mathcal{N}(\theta|\theta_0, \Sigma_\theta)
$$
$$
p(y|\theta) = \mathcal{N}(y| \Phi \theta, \Sigma_\eta)
$$
I would like to compute the posterior distribution,
$$
p(\theta|y) = \frac{p(y|\theta)p(\theta)}{p(y)}
$$
The easiest way I can think of doing this would be by maximizing the expression $p(y|\theta)p(\theta)$ with respect to theta to find the posterior mean and then calculating the MSE to find the posterior covariance matrix.
Is there another way of find the posterior distribution?
The reason I am asking is that when I found the posterior mean I ended up with the following expression:
$$
\hat\theta_{MAP} = (\Sigma_\theta^{-1} + \Phi^T\Sigma_\eta^{-1}\Phi)^{-1}(\Sigma_\theta^{-1}\theta_0 + \Phi^T\Sigma_\eta^{-1}y)
$$
Which is in a slightly different form than what I have found in my book.
$$
\hat\theta_{MAP} = \theta_0 + (\Sigma_\theta^{-1} + \Phi^T\Sigma_\eta^{-1}\Phi)^{-1}\Phi^T\Sigma_\eta^{-1}(y-\Phi\theta_0)
$$
These are both equivalent, but the different formulation makes me believe that they used a different method to find the posterior mean.
So I am wondering what methods are available to find a gaussian posterior? How could I derive the posterior mean as in the second form I showed above?
 A: We can ignore the term $(\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)^{-1}\Phi^{T}\Sigma_{\eta}^{-1}y$ that appears in both expressions, and show that the remaining terms:
$$(\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)^{-1}\Sigma_{\theta}^{-1}\theta_0$$
from the first equation and
$$\theta_0 - (\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)^{-1}\Phi^{T}\Sigma_{\eta}^{-1}\Phi\theta_0$$
from the second equation are equivalent.
Observe that
$$\theta_0 = (\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)^{-1}(\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)\theta_0$$
as the two matrices within the $()$ are inverses of each other, so multiply to the identity matrix.
We will simplify our notation in what follows by defining $A = (\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)^{-1}$.  Substituting for $\theta_0$ in the second expression gives us:
$$A(\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)\theta_0  - A\Phi^{T}\Sigma_{\eta}^{-1}\Phi\theta_0$$
Expanding the first term results in:
$$A\Sigma_{\theta}^{-1}\theta_0 + A\Phi^{T}\Sigma_{\eta}^{-1}\Phi\theta_0 -  A\Phi^{T}\Sigma_{\eta}^{-1}\Phi\theta_0$$
which evidently equals:
$$(\Sigma_{\theta}^{-1} + \Phi^{T}\Sigma_{\eta}^{-1}\Phi)^{-1}\Sigma_{\theta}^{-1}\theta_0$$
which is the term in the first expression.   Therefore the two terms are just algebraic rearrangements of each other.
You can derive the second term from the first by starting with the result and reversing the steps - adding $A\Phi^{T}\Sigma_{\eta}^{-1}\Phi\theta_0 -  A\Phi^{T}\Sigma_{\eta}^{-1}\Phi\theta_0$ to the second term, rearranging the expression, etc.
