3
$\begingroup$

Question: Assume the following classical linear normal regression model: \begin{gather*} y_{i} = \beta_1 x_{1i} + \beta_2 x_{2i} + \cdots + \beta_K x_{Ki} + e_i \\ \underbrace{\boldsymbol{y}}_{n \times 1} = \underbrace{\boldsymbol{X}}_{n \times K}\underbrace{\boldsymbol{\beta}}_{K \times 1} + \underbrace{\boldsymbol{e}}_{n \times 1} \end{gather*}

where $e_i \sim iid \ N(0, \sigma^2)$ for $i = 1, 2, \cdots, n$ and $x_{1i} = 1$ for $i = 1, 2, \cdots, n$

Define $\boldsymbol{\beta} = (\beta_1, \beta_2, \cdots, \beta_K)'$ and assume a noninformative prior of the form $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, derive the conditional posterior pdfs for $\boldsymbol{\beta}$ and $\sigma$, that is, $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ and $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$.


I can derive the conditional posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ but I am not sure how to derive it for $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$. The answer to the latter is:

$p(\sigma|\boldsymbol{\beta}, \mathbf{y})=\frac{2}{\Gamma\left(\frac{n}{2}\right)}\left(\frac{na}{2}\right)^{\frac{n}{2}} \frac{1}{\sigma^{n+1}}\exp\left[-\frac{na}{2\sigma^2}\right] $

where $a = \frac{1}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$


The following is my working to derive the posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$

We have: \begin{align*} p(\boldsymbol{e}|\sigma) & = \prod_{i=1}^n p(e_i) \\ & = \prod_{i=1}^n (2\pi \sigma^2)^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2} \boldsymbol{e'}\boldsymbol{e}\right] \end{align*} Since $p(\boldsymbol{y}) = p(\boldsymbol{e})\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right|$ and $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{e} \Longleftrightarrow \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta}$, then $\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} = \frac{\partial \boldsymbol{y'}}{\partial \boldsymbol{y}} = \boldsymbol{I}_{n}$. Hence, $\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right| = \left| \boldsymbol{I}_{n}\right| = 1$. So: \begin{align*} p(\boldsymbol{y}|\boldsymbol{\beta}, \sigma) & = p(\boldsymbol{e}) \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \ \ \text{since} \ \ \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta} \end{align*} Thus we have: $$L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \propto (\sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]$$ Given $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, we have: \begin{align*} p(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) & \propto L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \times p(\boldsymbol{\beta}, \sigma) \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}} \exp\left\{-\frac{1}{2\sigma^2}\left[(n-K)\widehat{\sigma}^2 + (\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \right\} \\ & \text{where} \ \ \widehat{\sigma}^2 = \frac{1}{n-K} (\boldsymbol{y} - \boldsymbol{X}\boldsymbol{b})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b}) \ \ \text{and} \ \ \boldsymbol{b} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y} \\ & = \underbrace{\frac{1}{\sigma^K} \exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right]}_{f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})} \underbrace{\frac{1}{\sigma^{n-K+1}} \exp\left[-\frac{(n-K)\widehat{\sigma}^2}{2\sigma^2}\right]}_{f^n(\sigma|\mathbf{y})} \end{align*} Note that $f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})$ is a kernel for $\boldsymbol{\beta}$, where $\boldsymbol{\beta} \sim MVN(\boldsymbol{b}, \sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1})$ and $f^n(\sigma|\mathbf{y})$ is a kernel for $\sigma$ where $\sigma \sim IG(v, \widehat{\sigma})$ and $v = n-K$, $\widehat{\sigma}^2 = \frac{1}{v} \left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$. Thus we have: $$p(\boldsymbol{\beta}, \sigma|\mathbf{y}) = p(\boldsymbol{\beta}|\sigma, \boldsymbol{y})p(\sigma|\mathbf{y})$$ where: \begin{gather*} p(\boldsymbol{\beta}|\sigma, \boldsymbol{y}) = (2\pi)^{-\frac{K}{2}}|\sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1}|^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \\ p(\sigma|\mathbf{y}) = \frac{2}{\Gamma\left(\frac{v}{2}\right)} \left(\frac{v\widehat{\sigma}^2}{2}\right)^{\frac{v}{2}}\frac{1}{\sigma^{v+1}}\exp\left[-\frac{v\widehat{\sigma}^2}{2\sigma^2}\right] \end{gather*}

$\endgroup$
  • $\begingroup$ I answered this question in my previous answer to your question stats.stackexchange.com/questions/68424/… by using the decomposition $$p(\beta,\sigma^2|y,X)=p(\beta|\sigma^2,y,X)p(\sigma^2|y,X)$$. $\endgroup$ – user25658 Aug 27 '13 at 16:53
  • 1
    $\begingroup$ @BakakP Thanks for your indepth post in the other thread, however here I am wondering how you can derive the conditional distribution of $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$. Similar to your other post, we both arrived at the same posterior pdf for $\sigma|\mathbf{y}$, but how do you derive the conditional pdf for $\sigma|\boldsymbol{\beta}, \mathbf{y}$? $\endgroup$ – TeTs Aug 27 '13 at 17:05
  • $\begingroup$ I don't derive $\sigma|\beta,y$ $\endgroup$ – user25658 Aug 27 '13 at 17:10
  • $\begingroup$ Once you have obtained $\beta,\sigma|y$ the easy thing to do is to integrate out $\beta$ to obtain $\sigma|y$, i.e., $$p(\sigma|y)=\int_{R^p}p(\beta,\sigma|y)d\beta$$ $\endgroup$ – user25658 Aug 27 '13 at 17:12
  • $\begingroup$ @BakakP Yes I understand, however this is a separate question that I'm wondering. How do we derive the distribution of $\sigma|\beta, \mathbf{y}$? I have no idea... $\endgroup$ – TeTs Aug 27 '13 at 17:15
2
$\begingroup$

We have the following posterior distribution

$$p(\beta,\sigma^2|y)\propto L(\beta,\sigma^2|y)\times p(\beta,\sigma^2)$$ and so, the full conditional $\sigma^2,\beta,y$ can be derived as the following:

\begin{align*} p(\sigma^2|\beta,y)&\propto\left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2}(y-X\beta)'(y-X\beta)\right\}\times\frac{1}{\sigma^2}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}\left[(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)+(n-k)\hat\sigma^2\right]\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)\right\}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto(\sigma^2)^{-(n/2+1)}\exp\left\{-\frac{1}{\sigma^2}\frac{(n-k)\hat\sigma^2}{2}\right\} \end{align*}

And so $$\sigma|\beta,y \sim\text{Inverse-Gamma}\left(\frac{n}{2},\frac{(n-k)\hat\sigma^2}{2}\right)$$ Double check that my arithmetic is all fine but that is how you would derive it.

$\endgroup$
  • $\begingroup$ I leave out the algebra between the first two lines of derivations but that is just decomposing the likelihood using the sufficient statistics. $\endgroup$ – user25658 Aug 27 '13 at 17:45
  • $\begingroup$ @BakakP Great, I understand all of that, however the only thing I am unclear about is this the statement: $p(\sigma^2|\beta,y)\propto\left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2} (y-X\beta)'(y-X\beta)\right\}\times\frac{1}{\sigma^2}$ ...(1). I know that $L(\beta,\sigma^2|y) \propto \left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2}(y-X\beta)'(y-X\beta)\right\}$ and the noninformative prior is given by $p(\beta,\sigma^2) \propto \frac{1}{\sigma^2}$ $\endgroup$ – TeTs Aug 27 '13 at 17:55
  • $\begingroup$ (cont...), but shouldn't the LHS of (1) be $p(\sigma^2, \beta|y)$ rather than $p(\sigma^2|\beta,y)$ since $p(\beta,\sigma^2|y)\propto L(\beta,\sigma^2|y)\times p(\beta,\sigma^2)$? $\endgroup$ – TeTs Aug 27 '13 at 17:56
  • $\begingroup$ It's the standard way to find a full conditional. You set the full conditional that you want (in your case $p(\sigma^2|\beta,y)$) equal to the full posterior and you start dropping terms where ever you can based on what you conditioned on. $\endgroup$ – user25658 Aug 27 '13 at 17:56
  • $\begingroup$ What exactly is meant by "full conditional?" $\endgroup$ – TeTs Aug 27 '13 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.