Bayesian Linear Model Posterior as Sum of Squares?

Question

As part of a homework, I am asked to do the math from the Normal-Inverse Gamma linear regression model.

Starting from priors $N(\beta_0, \sigma^2 A)$ and $IG(\alpha_0, \delta_0)$ and with the help of "Introduction to Bayesian Econometrics" by Edward Greenberg, I was able to find the posterior distributions for $\beta$ and $\sigma^2$, as $$N(\beta^*, \sigma^2 B)\text{ and }IG(\alpha_1, \delta_1)$$ respectively. The updated parameters are $$B = (A^{-1} + X^TX)^{-1},\ \ \alpha_1 = \alpha_0 + n,$$ $$\beta^* = B(A^{-1}\beta_0 + X^TX \hat{\beta})$$ and $$\delta_1 = \delta_0 + y^Ty + \beta_0^TA^{-1}\beta_0 - \beta^{*T}B^{-1}\beta^*$$ where $$\hat{\beta} = (X^TX)^{-1}X^Ty$$ is the Maximum Likelihood Estimate. From these expressions, I am asked to express $\delta_1$ as $$\delta_0 + (y-X\hat{\beta})'(y-X\hat{\beta}) + (\beta_0 - \hat{\beta})[(X^TX)^{-1} + A]^{-1}(\beta_0 - \hat{\beta}).$$ I have seen in a couple of books that they arrive to this expression but they never do the workout. Does anyone have any pointers on how I could arrive to the solution? Anything would be helpful at this point.

Answer 1

I am posting the method I used to solve this problem for future reference. In essence, it is solved by applying the matrix identity Christoph pointed out to me as well as a more elementary identity.

First, express $\delta_1$ as \begin{align*} \delta_1 & = \delta_0 + y^T y + \beta_0^T A^{-1}\beta_0 - {\beta^*}^T B^{-1}\beta^* \\ & = \delta_0 + y^T y + \beta_0^T A^{-1}\beta_0 - (A^{-1}\beta_0 + X^TX \hat{\beta}) ^T B (A^{-1}\beta_0 + X^TX \hat{\beta}) \\ & = \delta_0 + y^T y + \beta_0^T A^{-1}\beta_0 - \hat{\beta}^T X^TX B X^TX \hat{\beta} - 2\hat{\beta}^T X^TX BA^{-1}\beta_0 - \beta_0^T A^{-1}BA^{-1}\beta_0 \end{align*}

Adding and subtracting $\hat{\beta}^T X^TX \hat{\beta}$ and grouping like terms yields \begin{align*} \delta_1 = \delta_0 + y^T y - \hat{\beta}^T X^TX \hat{\beta} + \hat{\beta}^T (X^TX - X^TX B X^TX) \hat{\beta} - 2\hat{\beta}^T X^TX BA^{-1}\beta_0 + \beta_0^T (A^{-1} - A^{-1}BA^{-1})\beta_0 \end{align*}

Note that $$(y - X\hat{\beta})^T(y - X\hat{\beta}) = y^T y - 2 \hat{\beta}^T X^Ty + \hat{\beta}^T X^TX\hat{\beta} = y^T y - \hat{\beta}^T X^TX\hat{\beta}.$$

Let $C$ and $D$ be invertible matrices of equal dimension. I will make use of the following matrix identities: \begin{align*} (C + D)^{-1} & = C^{-1}(D^{-1} + C^{-1})^{-1}D^{-1} \\ & = C^{-1} - C^{-1}(D^{-1} + C^{-1})^{-1}C^{-1} \end{align*}

This gives three alternative ways to express $[(X^TX)^{-1} + A]^{-1}$: \begin{align*} [(X^TX)^{-1} + A]^{-1} & = X^TX - X^TX (X^TX + A^{-1})^{-1} X^TX \\ & = X^TX (X^TX + A^{-1})^{-1} A^{-1} \\ & = A^{-1} - A^{-1} (X^TX + A^{-1})^{-1} A^{-1} \end{align*}

Putting everything together gives \begin{align*} \delta_1 & = \delta_0 + (y - X\hat{\beta})^T(y - X\hat{\beta}) + \hat{\beta}^T [(X^TX)^{-1} + A]^{-1} \hat{\beta} - 2\hat{\beta}^T [(X^TX)^{-1} + A]^{-1} \beta_0 + \beta_0^T [(X^TX)^{-1} + A]^{-1}\beta_0 \\ & = \delta_0 + (y - X\hat{\beta})^T(y - X\hat{\beta}) + (\hat{\beta} - \beta_0)^T [(X^TX)^{-1} + A]^{-1} (\hat{\beta} - \beta_0) \end{align*}

as needed. Going one step futher, denote $\hat{\sigma}^2 = (y - X\hat{\beta})^T(y - X\hat{\beta})/n$ as the MLE for the variance. Then $$\delta_1 = \delta_0 + n\hat{\sigma}^2 + (\hat{\beta} - \beta_0)^T [(X^TX)^{-1} + A]^{-1} (\hat{\beta} - \beta_0).$$