5
$\begingroup$

Question: Assume the following classical linear normal regression model: \begin{gather*} y_{i} = \beta_1 x_{1i} + \beta_2 x_{2i} + \cdots + \beta_K x_{Ki} + e_i \\ \underbrace{\boldsymbol{y}}_{n \times 1} = \underbrace{\boldsymbol{X}}_{n \times K}\underbrace{\boldsymbol{\beta}}_{K \times 1} + \underbrace{\boldsymbol{e}}_{n \times 1} \end{gather*}

where $e_i \sim iid \ N(0, \sigma^2)$ for $i = 1, 2, \cdots, n$ and $x_{1i} = 1$ for $i = 1, 2, \cdots, n$

Define $\boldsymbol{\beta} = (\beta_1, \beta_2, \cdots, \beta_K)'$ and assume a noninformative prior of the form $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, derive the conditional posterior pdfs for $\boldsymbol{\beta}$ and $\sigma$, that is, $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ and $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$.


I can derive the conditional posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$ but I am not sure how to derive it for $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$. The answer to the latter is:

$p(\sigma|\boldsymbol{\beta}, \mathbf{y})=\frac{2}{\Gamma\left(\frac{n}{2}\right)}\left(\frac{na}{2}\right)^{\frac{n}{2}} \frac{1}{\sigma^{n+1}}\exp\left[-\frac{na}{2\sigma^2}\right] $

where $a = \frac{1}{n}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$


The following is my working to derive the posterior pdf for $p(\boldsymbol{\beta}|\sigma, \mathbf{y})$

We have: \begin{align*} p(\boldsymbol{e}|\sigma) & = \prod_{i=1}^n p(e_i) \\ & = \prod_{i=1}^n (2\pi \sigma^2)^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}}\exp\left[-\frac{1}{2\sigma^2}\sum_{i=1}^n e_i^2\right] \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2} \boldsymbol{e'}\boldsymbol{e}\right] \end{align*} Since $p(\boldsymbol{y}) = p(\boldsymbol{e})\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right|$ and $\boldsymbol{y} = \boldsymbol{X}\boldsymbol{\beta} + \boldsymbol{e} \Longleftrightarrow \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta}$, then $\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} = \frac{\partial \boldsymbol{y'}}{\partial \boldsymbol{y}} = \boldsymbol{I}_{n}$. Hence, $\left|\frac{\partial \boldsymbol{e'}}{\partial \boldsymbol{y}} \right| = \left| \boldsymbol{I}_{n}\right| = 1$. So: \begin{align*} p(\boldsymbol{y}|\boldsymbol{\beta}, \sigma) & = p(\boldsymbol{e}) \\ & = (2\pi \sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \ \ \text{since} \ \ \boldsymbol{e} = \boldsymbol{y} - \boldsymbol{X}\boldsymbol{\beta} \end{align*} Thus we have: $$L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \propto (\sigma^2)^{-\frac{n}{2}} \exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right]$$ Given $p(\boldsymbol{\beta}, \sigma) \propto \frac{1}{\sigma}$, we have: \begin{align*} p(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) & \propto L(\boldsymbol{\beta}, \sigma|\boldsymbol{y}) \times p(\boldsymbol{\beta}, \sigma) \\ & \propto \frac{1}{\sigma^n}\exp\left[-\frac{1}{2\sigma^2}\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)\right] \times \frac{1}{\sigma} \\ & = \frac{1}{\sigma^{n+1}} \exp\left\{-\frac{1}{2\sigma^2}\left[(n-K)\widehat{\sigma}^2 + (\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \right\} \\ & \text{where} \ \ \widehat{\sigma}^2 = \frac{1}{n-K} (\boldsymbol{y} - \boldsymbol{X}\boldsymbol{b})'(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{b}) \ \ \text{and} \ \ \boldsymbol{b} = (\boldsymbol{X}'\boldsymbol{X})^{-1}\boldsymbol{X}'\boldsymbol{y} \\ & = \underbrace{\frac{1}{\sigma^K} \exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right]}_{f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})} \underbrace{\frac{1}{\sigma^{n-K+1}} \exp\left[-\frac{(n-K)\widehat{\sigma}^2}{2\sigma^2}\right]}_{f^n(\sigma|\mathbf{y})} \end{align*} Note that $f^n(\boldsymbol{\beta}|\sigma, \boldsymbol{y})$ is a kernel for $\boldsymbol{\beta}$, where $\boldsymbol{\beta} \sim MVN(\boldsymbol{b}, \sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1})$ and $f^n(\sigma|\mathbf{y})$ is a kernel for $\sigma$ where $\sigma \sim IG(v, \widehat{\sigma})$ and $v = n-K$, $\widehat{\sigma}^2 = \frac{1}{v} \left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)'\left(\boldsymbol{y}-\boldsymbol{X}\boldsymbol{\beta}\right)$. Thus we have: $$p(\boldsymbol{\beta}, \sigma|\mathbf{y}) = p(\boldsymbol{\beta}|\sigma, \boldsymbol{y})p(\sigma|\mathbf{y})$$ where: \begin{gather*} p(\boldsymbol{\beta}|\sigma, \boldsymbol{y}) = (2\pi)^{-\frac{K}{2}}|\sigma^2(\boldsymbol{X}'\boldsymbol{X})^{-1}|^{-\frac{1}{2}}\exp\left[-\frac{1}{2\sigma^2}(\boldsymbol{\beta} - \boldsymbol{b})'\boldsymbol{X}'\boldsymbol{X}(\boldsymbol{\beta}-\boldsymbol{b})\right] \\ p(\sigma|\mathbf{y}) = \frac{2}{\Gamma\left(\frac{v}{2}\right)} \left(\frac{v\widehat{\sigma}^2}{2}\right)^{\frac{v}{2}}\frac{1}{\sigma^{v+1}}\exp\left[-\frac{v\widehat{\sigma}^2}{2\sigma^2}\right] \end{gather*}

$\endgroup$
6
  • $\begingroup$ I answered this question in my previous answer to your question stats.stackexchange.com/questions/68424/… by using the decomposition $$p(\beta,\sigma^2|y,X)=p(\beta|\sigma^2,y,X)p(\sigma^2|y,X)$$. $\endgroup$
    – user25658
    Aug 27, 2013 at 16:53
  • 1
    $\begingroup$ @BakakP Thanks for your indepth post in the other thread, however here I am wondering how you can derive the conditional distribution of $p(\sigma|\boldsymbol{\beta}, \mathbf{y})$. Similar to your other post, we both arrived at the same posterior pdf for $\sigma|\mathbf{y}$, but how do you derive the conditional pdf for $\sigma|\boldsymbol{\beta}, \mathbf{y}$? $\endgroup$
    – TeTs
    Aug 27, 2013 at 17:05
  • $\begingroup$ I don't derive $\sigma|\beta,y$ $\endgroup$
    – user25658
    Aug 27, 2013 at 17:10
  • $\begingroup$ Once you have obtained $\beta,\sigma|y$ the easy thing to do is to integrate out $\beta$ to obtain $\sigma|y$, i.e., $$p(\sigma|y)=\int_{R^p}p(\beta,\sigma|y)d\beta$$ $\endgroup$
    – user25658
    Aug 27, 2013 at 17:12
  • $\begingroup$ @BakakP Yes I understand, however this is a separate question that I'm wondering. How do we derive the distribution of $\sigma|\beta, \mathbf{y}$? I have no idea... $\endgroup$
    – TeTs
    Aug 27, 2013 at 17:15

1 Answer 1

2
$\begingroup$

We have the following posterior distribution

$$p(\beta,\sigma^2|y)\propto L(\beta,\sigma^2|y)\times p(\beta,\sigma^2)$$ and so, the full conditional $\sigma^2,\beta,y$ can be derived as the following:

\begin{align*} p(\sigma^2|\beta,y)&\propto\left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2}(y-X\beta)'(y-X\beta)\right\}\times\frac{1}{\sigma^2}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}\left[(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)+(n-k)\hat\sigma^2\right]\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(\beta-\hat\beta)'(X'X)(\beta-\hat\beta)\right\}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto\left(\frac{1}{\sigma^2}\right)^{n/2+1}\exp\left\{-\frac{1}{2\sigma^2}(n-k)\hat\sigma^2\right\}\\ &\propto(\sigma^2)^{-(n/2+1)}\exp\left\{-\frac{1}{\sigma^2}\frac{(n-k)\hat\sigma^2}{2}\right\} \end{align*}

And so $$\sigma|\beta,y \sim\text{Inverse-Gamma}\left(\frac{n}{2},\frac{(n-k)\hat\sigma^2}{2}\right)$$ Double check that my arithmetic is all fine but that is how you would derive it.

$\endgroup$
9
  • $\begingroup$ I leave out the algebra between the first two lines of derivations but that is just decomposing the likelihood using the sufficient statistics. $\endgroup$
    – user25658
    Aug 27, 2013 at 17:45
  • $\begingroup$ @BakakP Great, I understand all of that, however the only thing I am unclear about is this the statement: $p(\sigma^2|\beta,y)\propto\left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2} (y-X\beta)'(y-X\beta)\right\}\times\frac{1}{\sigma^2}$ ...(1). I know that $L(\beta,\sigma^2|y) \propto \left(\frac{1}{\sigma^2}\right)^{n/2}\exp\left\{-\frac{1}{2\sigma^2}(y-X\beta)'(y-X\beta)\right\}$ and the noninformative prior is given by $p(\beta,\sigma^2) \propto \frac{1}{\sigma^2}$ $\endgroup$
    – TeTs
    Aug 27, 2013 at 17:55
  • $\begingroup$ (cont...), but shouldn't the LHS of (1) be $p(\sigma^2, \beta|y)$ rather than $p(\sigma^2|\beta,y)$ since $p(\beta,\sigma^2|y)\propto L(\beta,\sigma^2|y)\times p(\beta,\sigma^2)$? $\endgroup$
    – TeTs
    Aug 27, 2013 at 17:56
  • $\begingroup$ It's the standard way to find a full conditional. You set the full conditional that you want (in your case $p(\sigma^2|\beta,y)$) equal to the full posterior and you start dropping terms where ever you can based on what you conditioned on. $\endgroup$
    – user25658
    Aug 27, 2013 at 17:56
  • $\begingroup$ What exactly is meant by "full conditional?" $\endgroup$
    – TeTs
    Aug 27, 2013 at 17:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.