# Deriving posterior pdf in classical linear normal regression model under noninformative prior

Question: Assume the following classical linear normal regression model: \begin{gather*} y_{i} = \beta_1 x_{1i} + \beta_2 x_{2i} + \cdots + \beta_K x_{Ki} + e_i \\ \underbrace{\boldsymbol{y}}_{n \times 1} = \underbrace{\boldsymbol{X}}_{n \times K}\underbrace{\boldsymbol{\beta}}_{K \times 1} + \underbrace{\boldsymbol{e}}_{n \times 1} \end{gather*}

where $e_i \sim iid \ N(0, \sigma^2)$ for $i = 1, 2, \cdots, n$ and $x_{1i} = 1$ for $i = 1, 2, \cdots, n$

Define $\boldsymbol{\beta} = (\beta_1, \beta_2, \cdots, \beta_K)'$ and assume a noninformative prior of the form $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, derive the conditional posterior pdfs for $\boldsymbol{\beta}$ and $\sigma$, that is, $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ and $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$.

I can derive the conditional posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ but I am not sure how to derive it for $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$. The answer to the latter is:

$p(\sigma|\boldsymbol{\beta}, \mathbf{y})=\frac{2}{\Gamma\left(\frac{n}{2}\right)}\left(\frac{na}{2}\right)^{\frac{n}{2}} \frac{1}{\sigma^{n+1}}\exp\left[-\frac{na}{2\sigma^2}\right]$

where $a = \frac{1}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$

The following is my working to derive the posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$

We have: \begin{align*} p(\boldsymbol{e}|\sigma) & = \prod_{i=1}^n p(e_i) \\ & = \prod_{i=1}^n (2\pi \sigma^2)^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2} \boldsymbol{e'}\boldsymbol{e}\right] \end{align*} Since $p(\boldsymbol{y}) = p(\boldsymbol{e})\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right|$ and $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{e} \Longleftrightarrow \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta}$, then $\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} = \frac{\partial \boldsymbol{y'}}{\partial \boldsymbol{y}} = \boldsymbol{I}_{n}$. Hence, $\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right| = \left| \boldsymbol{I}_{n}\right| = 1$. So: \begin{align*} p(\boldsymbol{y}|\boldsymbol{\beta}, \sigma) & = p(\boldsymbol{e}) \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \ \ \text{since} \ \ \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta} \end{align*} Thus we have: $$L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \propto (\sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]$$ Given $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, we have: \begin{align*} p(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) & \propto L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \times p(\boldsymbol{\beta}, \sigma) \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}} \exp\left\{-\frac{1}{2\sigma^2}\left[(n-K)\widehat{\sigma}^2 + (\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \right\} \\ & \text{where} \ \ \widehat{\sigma}^2 = \frac{1}{n-K} (\boldsymbol{y} - \boldsymbol{X}\boldsymbol{b})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b}) \ \ \text{and} \ \ \boldsymbol{b} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y} \\ & = \underbrace{\frac{1}{\sigma^K} \exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right]}_{f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})} \underbrace{\frac{1}{\sigma^{n-K+1}} \exp\left[-\frac{(n-K)\widehat{\sigma}^2}{2\sigma^2}\right]}_{f^n(\sigma|\mathbf{y})} \end{align*} Note that $f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})$ is a kernel for $\boldsymbol{\beta}$, where $\boldsymbol{\beta} \sim MVN(\boldsymbol{b}, \sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1})$ and $f^n(\sigma|\mathbf{y})$ is a kernel for $\sigma$ where $\sigma \sim IG(v, \widehat{\sigma})$ and $v = n-K$, $\widehat{\sigma}^2 = \frac{1}{v} \left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$. Thus we have: $$p(\boldsymbol{\beta}, \sigma|\mathbf{y}) = p(\boldsymbol{\beta}|\sigma, \boldsymbol{y})p(\sigma|\mathbf{y})$$ where: \begin{gather*} p(\boldsymbol{\beta}|\sigma, \boldsymbol{y}) = (2\pi)^{-\frac{K}{2}}|\sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1}|^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \\ p(\sigma|\mathbf{y}) = \frac{2}{\Gamma\left(\frac{v}{2}\right)} \left(\frac{v\widehat{\sigma}^2}{2}\right)^{\frac{v}{2}}\frac{1}{\sigma^{v+1}}\exp\left[-\frac{v\widehat{\sigma}^2}{2\sigma^2}\right] \end{gather*}

• I answered this question in my previous answer to your question stats.stackexchange.com/questions/68424/… by using the decomposition $$p(\beta,\sigma^2|y,X)=p(\beta|\sigma^2,y,X)p(\sigma^2|y,X)$$. – user25658 Aug 27 '13 at 16:53
• @BakakP Thanks for your indepth post in the other thread, however here I am wondering how you can derive the conditional distribution of $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$. Similar to your other post, we both arrived at the same posterior pdf for $\sigma|\mathbf{y}$, but how do you derive the conditional pdf for $\sigma|\boldsymbol{\beta}, \mathbf{y}$? – TeTs Aug 27 '13 at 17:05
• I don't derive $\sigma|\beta,y$ – user25658 Aug 27 '13 at 17:10
• Once you have obtained $\beta,\sigma|y$ the easy thing to do is to integrate out $\beta$ to obtain $\sigma|y$, i.e., $$p(\sigma|y)=\int_{R^p}p(\beta,\sigma|y)d\beta$$ – user25658 Aug 27 '13 at 17:12
• @BakakP Yes I understand, however this is a separate question that I'm wondering. How do we derive the distribution of $\sigma|\beta, \mathbf{y}$? I have no idea... – TeTs Aug 27 '13 at 17:15

We have the following posterior distribution

$$p(\beta,\sigma^2|y)\propto L(\beta,\sigma^2|y)\times p(\beta,\sigma^2)$$ and so, the full conditional $\sigma^2,\beta,y$ can be derived as the following:

\begin{align*} p(\sigma^2|\beta,y)&\propto\left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2}(y-X\beta)'(y-X\beta)\right\}\times\frac{1}{\sigma^2}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}\left[(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)+(n-k)\hat\sigma^2\right]\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)\right\}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto(\sigma^2)^{-(n/2+1)}\exp\left\{-\frac{1}{\sigma^2}\frac{(n-k)\hat\sigma^2}{2}\right\} \end{align*}

And so $$\sigma|\beta,y \sim\text{Inverse-Gamma}\left(\frac{n}{2},\frac{(n-k)\hat\sigma^2}{2}\right)$$ Double check that my arithmetic is all fine but that is how you would derive it.

• I leave out the algebra between the first two lines of derivations but that is just decomposing the likelihood using the sufficient statistics. – user25658 Aug 27 '13 at 17:45
• @BakakP Great, I understand all of that, however the only thing I am unclear about is this the statement: $p(\sigma^2|\beta,y)\propto\left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2} (y-X\beta)'(y-X\beta)\right\}\times\frac{1}{\sigma^2}$ ...(1). I know that $L(\beta,\sigma^2|y) \propto \left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2}(y-X\beta)'(y-X\beta)\right\}$ and the noninformative prior is given by $p(\beta,\sigma^2) \propto \frac{1}{\sigma^2}$ – TeTs Aug 27 '13 at 17:55
• (cont...), but shouldn't the LHS of (1) be $p(\sigma^2, \beta|y)$ rather than $p(\sigma^2|\beta,y)$ since $p(\beta,\sigma^2|y)\propto L(\beta,\sigma^2|y)\times p(\beta,\sigma^2)$? – TeTs Aug 27 '13 at 17:56
• It's the standard way to find a full conditional. You set the full conditional that you want (in your case $p(\sigma^2|\beta,y)$) equal to the full posterior and you start dropping terms where ever you can based on what you conditioned on. – user25658 Aug 27 '13 at 17:56
• What exactly is meant by "full conditional?" – TeTs Aug 27 '13 at 17:58