# linear minimum mean squared error estimate under Gaussian prior

I am learning estimation theory through Steven M.Kay's book Fundamentals Of Statistical Signal Processing--Estimation Theory. In the Chapter 12 (Linear Bayesian Estimator), Theorem 12.1 (Bayesian Gauss-Markov Theorem) gives the LMMSE estimation of the signal based on the linear noisy measurement under the Gaussian prior assumption:

If the data are described by the Bayesian linear model form $$$$\boldsymbol{x}=\boldsymbol{H\theta}+\boldsymbol{w} \tag{12.25}$$$$ where $$\boldsymbol{x}$$ is an $$N \times 1$$ data vector, $$\boldsymbol{H}$$ is a known $$N\times p$$ observation matrix, $$\boldsymbol{\theta}$$ is a $$p \times 1$$ random vector of parameters whose realization is to be estimated and has mean $$E(\boldsymbol{\theta})$$ and covariance matrix $$\boldsymbol{C}_{\theta\theta}$$, and $$\boldsymbol{w}$$ is an $$N \times 1$$ random vector with zero mean and covariance matrix $$\boldsymbol{C}_w$$ and is uncorrelated with $$\boldsymbol{\theta}$$ (the joint PDF $$p(\boldsymbol{w},\boldsymbol{\theta})$$ is otherwise arbitrary), then the LMMSE estimator of $$\boldsymbol{\theta}$$ is \begin{align} \hat{\boldsymbol{\theta}} & = E(\boldsymbol{\theta})+\boldsymbol{C}_{\theta\theta}\boldsymbol{H}^T(\boldsymbol{H}\boldsymbol{C}_{\theta\theta}\boldsymbol{H}^T+\boldsymbol{C}_w)^{-1}(\boldsymbol{x}-\boldsymbol{H}E(\boldsymbol{\theta})) \tag{12.26} \\ & = E(\boldsymbol{\theta})+(\boldsymbol{C}_{\theta\theta}^{-1}+\boldsymbol{H}^T\boldsymbol{C}_w^{-1}\boldsymbol{H})^{-1}\boldsymbol{H}^T\boldsymbol{C}_w^{-1}(\boldsymbol{x}-\boldsymbol{H}E(\boldsymbol{\theta})) \tag{12.27} \end{align} The performance of the estimatior is measured by the error $$\boldsymbol{\epsilon}=\boldsymbol{\theta}-\hat{\boldsymbol{\theta}}$$ whose mean is zero and whose covariance matrix is \begin{align} \boldsymbol{C}_\boldsymbol{\epsilon} &= E_{\boldsymbol{x},\boldsymbol{\theta}}(\boldsymbol{\epsilon}\boldsymbol{\epsilon}^T) \\ & = \boldsymbol{C}_{\theta\theta} - \boldsymbol{C}_{\theta\theta}\boldsymbol{H}^T(\boldsymbol{H}\boldsymbol{C}_{\theta\theta}\boldsymbol{H}^T+\boldsymbol{C}_w)^{-1}\boldsymbol{H}\boldsymbol{C}_{\theta\theta} \tag{12.28} \\ & = (\boldsymbol{C}_{\theta\theta}^{-1}+\boldsymbol{H}^T\boldsymbol{C}_w^{-1}\boldsymbol{H})^{-1} \tag{12.29} \end{align}

Since the prior of $$\boldsymbol{\theta}$$ is Gaussian, the LMMSE estimate $$\hat{\boldsymbol{\theta}}_{LMMSE}$$ is equivalent to the MMSE estimate $$\hat{\boldsymbol{\theta}}_{MMSE}$$, and $$\hat{\boldsymbol{\theta}}_{MMSE}$$ is equal to the posterior mearn $$E(\boldsymbol{\theta}|\boldsymbol{x})$$. Since the prior and likelihood are both Gaussian, the posterior distribution $$p(\boldsymbol{\theta}|\boldsymbol{x})$$ is also Gaussian.

Here I am trying to derive $$\hat{\boldsymbol{\theta}}_{MMSE}$$ and $$\boldsymbol{C}_\boldsymbol{\epsilon}$$ from the perspective of PDF multiplication, that is, calculate $$p(\boldsymbol{\theta}|\boldsymbol{x}) \propto p(\boldsymbol{x}|\boldsymbol{\theta})p(\boldsymbol{\theta})=\mathcal{N}(\boldsymbol{x};\boldsymbol{H\theta},\boldsymbol{C}_{w})\mathcal{N}(\boldsymbol{\theta};E(\boldsymbol{\theta}),\boldsymbol{C}_{\theta\theta})$$, and formulate the quadratic and firse-order terms of $$\boldsymbol{\theta}$$ at the exponential to form a Gaussian PDF. The covariance matrix of $$p(\boldsymbol{\theta}|\boldsymbol{x})$$ I got matches 12.29, but the posterior mean is the following form: $$$$E(\boldsymbol{\theta}|\boldsymbol{x}) = \boldsymbol{C}_{\boldsymbol{\epsilon}}(\boldsymbol{H}^T\boldsymbol{C}_w^{-1} \boldsymbol{x}+\boldsymbol{C}_{\theta\theta}^{-1}E(\boldsymbol{\theta})) \tag{q1}$$$$

So my question is, is the posterior mean I got in q1 equal to the $$\hat{\boldsymbol{\theta}}$$ given in 12.26 and 12.27? If so, how can I reach that?

By the way, I can't find the way from 12.26 to 12.27 (12.28 to 12.29 either). So can someone give me a hint?