2
$\begingroup$

As part of a homework, I am asked to do the math from the Normal-Inverse Gamma linear regression model.

Starting from priors $N(\beta_0, \sigma^2 A)$ and $IG(\alpha_0, \delta_0)$ and with the help of "Introduction to Bayesian Econometrics" by Edward Greenberg, I was able to find the posterior distributions for $\beta$ and $\sigma^2$, as $$N(\beta^*, \sigma^2 B)\text{ and }IG(\alpha_1, \delta_1)$$ respectively. The updated parameters are $$B = (A^{-1} + X^TX)^{-1},\ \ \alpha_1 = \alpha_0 + n,$$ $$\beta^* = B(A^{-1}\beta_0 + X^TX \hat{\beta})$$ and $$\delta_1 = \delta_0 + y^Ty + \beta_0^TA^{-1}\beta_0 - \beta^{*T}B^{-1}\beta^*$$ where $$\hat{\beta} = (X^TX)^{-1}X^Ty$$ is the Maximum Likelihood Estimate. From these expressions, I am asked to express $\delta_1$ as $$\delta_0 + (y-X\hat{\beta})'(y-X\hat{\beta}) + (\beta_0 - \hat{\beta})[(X^TX)^{-1} + A]^{-1}(\beta_0 - \hat{\beta}).$$ I have seen in a couple of books that they arrive to this expression but they never do the workout. Does anyone have any pointers on how I could arrive to the solution? Anything would be helpful at this point.

$\endgroup$
1
  • $\begingroup$ Possible lines of attack: 1. You can write $(y-X\hat{\beta})'(y-X\hat{\beta})$ as $$(y-X\hat{\beta})'(y-X\hat{\beta})=(y-X\beta)'(y-X\beta)-(\beta-\hat\beta)'X'X(\beta-\hat\beta) $$ 2. You can rewrite $((X'X)^{-1}+A)^{-1}$ as $A^{-1}(I-BA^{-1})$ using this. $\endgroup$ Mar 6 '15 at 11:44
1
$\begingroup$

I am posting the method I used to solve this problem for future reference. In essence, it is solved by applying the matrix identity Christoph pointed out to me as well as a more elementary identity.

First, express $\delta_1$ as \begin{align*} \delta_1 & = \delta_0 + y^T y + \beta_0^T A^{-1}\beta_0 - {\beta^*}^T B^{-1}\beta^* \\ & = \delta_0 + y^T y + \beta_0^T A^{-1}\beta_0 - (A^{-1}\beta_0 + X^TX \hat{\beta}) ^T B (A^{-1}\beta_0 + X^TX \hat{\beta}) \\ & = \delta_0 + y^T y + \beta_0^T A^{-1}\beta_0 - \hat{\beta}^T X^TX B X^TX \hat{\beta} - 2\hat{\beta}^T X^TX BA^{-1}\beta_0 - \beta_0^T A^{-1}BA^{-1}\beta_0 \end{align*}

Adding and subtracting $\hat{\beta}^T X^TX \hat{\beta}$ and grouping like terms yields \begin{align*} \delta_1 = \delta_0 + y^T y - \hat{\beta}^T X^TX \hat{\beta} + \hat{\beta}^T (X^TX - X^TX B X^TX) \hat{\beta} - 2\hat{\beta}^T X^TX BA^{-1}\beta_0 + \beta_0^T (A^{-1} - A^{-1}BA^{-1})\beta_0 \end{align*}

Note that $$(y - X\hat{\beta})^T(y - X\hat{\beta}) = y^T y - 2 \hat{\beta}^T X^Ty + \hat{\beta}^T X^TX\hat{\beta} = y^T y - \hat{\beta}^T X^TX\hat{\beta}.$$

Let $C$ and $D$ be invertible matrices of equal dimension. I will make use of the following matrix identities: \begin{align*} (C + D)^{-1} & = C^{-1}(D^{-1} + C^{-1})^{-1}D^{-1} \\ & = C^{-1} - C^{-1}(D^{-1} + C^{-1})^{-1}C^{-1} \end{align*}

This gives three alternative ways to express $[(X^TX)^{-1} + A]^{-1}$: \begin{align*} [(X^TX)^{-1} + A]^{-1} & = X^TX - X^TX (X^TX + A^{-1})^{-1} X^TX \\ & = X^TX (X^TX + A^{-1})^{-1} A^{-1} \\ & = A^{-1} - A^{-1} (X^TX + A^{-1})^{-1} A^{-1} \end{align*}

Putting everything together gives \begin{align*} \delta_1 & = \delta_0 + (y - X\hat{\beta})^T(y - X\hat{\beta}) + \hat{\beta}^T [(X^TX)^{-1} + A]^{-1} \hat{\beta} - 2\hat{\beta}^T [(X^TX)^{-1} + A]^{-1} \beta_0 + \beta_0^T [(X^TX)^{-1} + A]^{-1}\beta_0 \\ & = \delta_0 + (y - X\hat{\beta})^T(y - X\hat{\beta}) + (\hat{\beta} - \beta_0)^T [(X^TX)^{-1} + A]^{-1} (\hat{\beta} - \beta_0) \end{align*}

as needed. Going one step futher, denote $\hat{\sigma}^2 = (y - X\hat{\beta})^T(y - X\hat{\beta})/n$ as the MLE for the variance. Then $$\delta_1 = \delta_0 + n\hat{\sigma}^2 + (\hat{\beta} - \beta_0)^T [(X^TX)^{-1} + A]^{-1} (\hat{\beta} - \beta_0).$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.