Deriving posterior pdf in classical linear normal regression model under noninformative prior

Question

Question: Assume the following classical linear normal regression model: \begin{gather*} y_{i} = \beta_1 x_{1i} + \beta_2 x_{2i} + \cdots + \beta_K x_{Ki} + e_i \\ \underbrace{\boldsymbol{y}}_{n \times 1} = \underbrace{\boldsymbol{X}}_{n \times K}\underbrace{\boldsymbol{\beta}}_{K \times 1} + \underbrace{\boldsymbol{e}}_{n \times 1} \end{gather*}

where $e_i \sim iid \ N(0, \sigma^2)$ for $i = 1, 2, \cdots, n$ and $x_{1i} = 1$ for $i = 1, 2, \cdots, n$

Define $\boldsymbol{\beta} = (\beta_1, \beta_2, \cdots, \beta_K)'$ and assume a noninformative prior of the form $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, derive the conditional posterior pdfs for $\boldsymbol{\beta}$ and $\sigma$, that is, $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ and $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$.

---

I can derive the conditional posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ but I am not sure how to derive it for $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$. The answer to the latter is:

$p(\sigma|\boldsymbol{\beta}, \mathbf{y})=\frac{2}{\Gamma\left(\frac{n}{2}\right)}\left(\frac{na}{2}\right)^{\frac{n}{2}} \frac{1}{\sigma^{n+1}}\exp\left[-\frac{na}{2\sigma^2}\right] $

where $a = \frac{1}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$

---

The following is my working to derive the posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$

We have: \begin{align*} p(\boldsymbol{e}|\sigma) & = \prod_{i=1}^n p(e_i) \\ & = \prod_{i=1}^n (2\pi \sigma^2)^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2} \boldsymbol{e'}\boldsymbol{e}\right] \end{align*} Since $p(\boldsymbol{y}) = p(\boldsymbol{e})\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right|$ and $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{e} \Longleftrightarrow \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta}$, then $\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} = \frac{\partial \boldsymbol{y'}}{\partial \boldsymbol{y}} = \boldsymbol{I}_{n}$. Hence, $\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right| = \left| \boldsymbol{I}_{n}\right| = 1$. So: \begin{align*} p(\boldsymbol{y}|\boldsymbol{\beta}, \sigma) & = p(\boldsymbol{e}) \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \ \ \text{since} \ \ \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta} \end{align*} Thus we have: $$L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \propto (\sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]$$ Given $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, we have: \begin{align*} p(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) & \propto L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \times p(\boldsymbol{\beta}, \sigma) \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}} \exp\left\{-\frac{1}{2\sigma^2}\left[(n-K)\widehat{\sigma}^2 + (\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \right\} \\ & \text{where} \ \ \widehat{\sigma}^2 = \frac{1}{n-K} (\boldsymbol{y} - \boldsymbol{X}\boldsymbol{b})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b}) \ \ \text{and} \ \ \boldsymbol{b} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y} \\ & = \underbrace{\frac{1}{\sigma^K} \exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right]}_{f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})} \underbrace{\frac{1}{\sigma^{n-K+1}} \exp\left[-\frac{(n-K)\widehat{\sigma}^2}{2\sigma^2}\right]}_{f^n(\sigma|\mathbf{y})} \end{align*} Note that $f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})$ is a kernel for $\boldsymbol{\beta}$, where $\boldsymbol{\beta} \sim MVN(\boldsymbol{b}, \sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1})$ and $f^n(\sigma|\mathbf{y})$ is a kernel for $\sigma$ where $\sigma \sim IG(v, \widehat{\sigma})$ and $v = n-K$, $\widehat{\sigma}^2 = \frac{1}{v} \left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$. Thus we have: $$p(\boldsymbol{\beta}, \sigma|\mathbf{y}) = p(\boldsymbol{\beta}|\sigma, \boldsymbol{y})p(\sigma|\mathbf{y})$$ where: \begin{gather*} p(\boldsymbol{\beta}|\sigma, \boldsymbol{y}) = (2\pi)^{-\frac{K}{2}}|\sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1}|^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \\ p(\sigma|\mathbf{y}) = \frac{2}{\Gamma\left(\frac{v}{2}\right)} \left(\frac{v\widehat{\sigma}^2}{2}\right)^{\frac{v}{2}}\frac{1}{\sigma^{v+1}}\exp\left[-\frac{v\widehat{\sigma}^2}{2\sigma^2}\right] \end{gather*}

Answer 1

We have the following posterior distribution

$$p(\beta,\sigma^2|y)\propto L(\beta,\sigma^2|y)\times p(\beta,\sigma^2)$$ and so, the full conditional $\sigma^2,\beta,y$ can be derived as the following:

\begin{align*} p(\sigma^2|\beta,y)&\propto\left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2}(y-X\beta)'(y-X\beta)\right\}\times\frac{1}{\sigma^2}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}\left[(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)+(n-k)\hat\sigma^2\right]\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)\right\}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto(\sigma^2)^{-(n/2+1)}\exp\left\{-\frac{1}{\sigma^2}\frac{(n-k)\hat\sigma^2}{2}\right\} \end{align*}

And so $$\sigma|\beta,y \sim\text{Inverse-Gamma}\left(\frac{n}{2},\frac{(n-k)\hat\sigma^2}{2}\right)$$ Double check that my arithmetic is all fine but that is how you would derive it.