# likelihood in bayesian linear regression

I was going through the derivation for the likelihood of Bayesian linear regression http://en.wikipedia.org/wiki/Bayesian_linear_regression#Posterior_distribution

I did not understand this step where the author explains the Conjugate prior distribution.

$(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})^T$ $(\mathbf{y} - \mathbf{X}\boldsymbol{\beta})$ =

$(\mathbf{y} - \mathbf{X}\boldsymbol{\hat\beta})^T$ $(\mathbf{y} - \mathbf{X}\boldsymbol{\hat\beta})$ + $(\mathbf{\beta} -\mathbf{\hat\beta})^T$$(\mathbf{X}^T \mathbf{X})$$(\mathbf{\beta} -\mathbf{\hat\beta})$

Need help breaking this down.

There a couple of passages that are tricky in this calculation. First:

$$(y - X\beta)^T \,(y - X\beta) = y^Ty - y^TX\beta - \beta^TX^Ty + \beta^TX^TX\beta$$

We have that $X$ is a $n \times k$ matrix, $\beta$ is $k \times 1$ and $y$ is $n \times 1$. So $y^TX\beta$ is a scalar. Hence, $y^TX\beta = (y^TX\beta)^T = \beta^TX^Ty$. Therefore:

$$(y - X\beta)^T \,(y - X\beta) = y^Ty - 2y^TX\beta + \beta^TX^TX\beta$$

Let's add and subtract the term $\widehat{\beta}^TX^TX\widehat{\beta}$ into this equation. We obtain then:

\begin{align} (y - X\beta)^T \,(y - X\beta) &= y^Ty - 2y^TX\beta + \beta^TX^TX\beta \pm \widehat{\beta}^TX^TX\widehat{\beta} \\ &= (y^Ty - \widehat{\beta}^TX^TX\widehat{\beta}) + (- 2y^TX\beta + \beta^TX^TX\beta + \widehat{\beta}^TX^TX\widehat{\beta}) \end{align}

In the second passage I've just separated the two terms. Let's first consider that:

$$\widehat{\beta} = (X^TX)^{-1} X^Ty$$

On top of that:

$$\widehat{\beta}^TX^TX\widehat{\beta} = \big((X^TX)^{-1} X^Ty\big)^T X^TX\widehat{\beta} = y^TX\big((X^TX)^T\big)^{-1}X^TX\widehat{\beta}$$

Now, since that $X^TX$ is symmetric $(X^TX)^T = X^TX$ so:

$$\widehat{\beta}^TX^TX\widehat{\beta} = y^TX(X^TX)^{-1}X^TX\widehat{\beta} = y^TX\widehat{\beta}$$

Let's consider the first term:

\begin{align} y^Ty - \widehat{\beta}^TX^TX\widehat{\beta} &= y^Ty - 2\widehat{\beta}^TX^TX\widehat{\beta} + \widehat{\beta}^TX^TX\widehat{\beta}\\ &= y^Ty - 2y^TX\widehat{\beta} + \widehat{\beta}^TX^TX\widehat{\beta}\\ &= (y - X\widehat{\beta})^T(y - X\widehat{\beta}) \end{align}

The last passage is done again because $y^TX\widehat{\beta}$ is a scalar.

For what concerns the second term I'll just show that the second term you need to obtain is equal to that. So:

\begin{align} (\beta - \widehat{\beta})^TX^TX(\beta - \widehat{\beta}) &= \beta^TX^TX(\beta - \widehat{\beta})- \widehat{\beta}^TX^TX(\beta - \widehat{\beta}) \\ &= \beta^TX^TX\beta - \beta^TX^TX\widehat{\beta} - \widehat{\beta}^TX^TX\beta + \widehat{\beta}^TX^TX\widehat{\beta}\\ &= \beta^TX^TX\beta - \beta^T X^Ty - y^TX\beta + \widehat{\beta}^TX^TX\widehat{\beta}\\ &= -2y^TX\beta + \beta^TX^TX\beta + \widehat{\beta}^TX^TX\widehat{\beta} \end{align}

The last passage, once again, because $y^TX\beta$ is a scalar.

• this is perfect – Ernest Presley Nov 29 '14 at 7:36