3
$\begingroup$

One of my homework problems for an intro Bayesian class is to derive the posterior of $\vec{\beta}$, for a simple linear regression problem. That is, given:

$Y = X\vec{\beta} + \epsilon$ for:

$Y$ is the response vector (nx1) for n many cases

$X$ is the regressor matrix (nxp) for n many cases and p many regressors

$\vec{\beta}$ is the coefficient vector (nx1)

$\epsilon$ is the error error term ~ N($\vec{0}$,$\sigma ^2I)$

The prior P($\vec{\beta}$) $\propto$ 1 (That is, the flat or reference prior).

Using the pdf for a multivar normal distribution, I started here:

$L(\vec{\beta}|X,\sigma^2)\propto 1/\sigma*e^{(-1/2\sigma^2)(\vec{y} - X\vec{\beta})'(\vec{y} - X\vec{\beta})}$

Focusing on the matrix algebra in the exponent:

$(\vec{y}'-\vec{\beta}'X')(\vec{y} - X\vec{\beta})$

$\vec{y}'\vec{y} - \vec{\beta}'X'\vec{y} - \vec{y}'X\vec{\beta} + \vec{\beta}'(X'X)\vec{\beta}$

I'm not sure where to go from here. Instead of completing the square, my professor usually has us get the exponent into the form:

$ax^2 - 2bx + c$ for x is the normally distributed variable, a,b,c do not contain x.

And we are told that a = $variance^{-1}$, and b = $mean/variance$ in order to figure out our kernel.

For this problem, I'm unable to get it into this form, and would appreciate any help in doing so, even if you just point me to the relevant matrix algebra rules. I'm an undergrad, and so deathly afraid of matrices, and working to overcome my fear. I've seen some related questions which gave the answer, but chose to ask for the following reasons: none that I've seen mention a flat prior, and none go over the matrix algebra in sufficient detail.

I'm going to be hanging around here all night, and am more than happy to address questions or concerns.

Thank you for your time and attention.

Improve this question
$\endgroup$
6
  • $\begingroup$ Thank you for your reply, Glen. It's true that complete the square and this method would lead to the same result, but my professor is adamant that these two methods are different, and that this one is better. I'm afraid I don't understand the second part of your comment: I thought $X$ was a nxp matrix, and so would not be its own transpose? Please excuse my ignorance of matrix algebra. I also should have been more clear on the notation: in the ax2 - 2bx + c form, x is the variable, not the X matrix. As I understand, because we're looking for a posterior on beta, we should be looking for a... $\endgroup$
    – John Madden
    Commented Sep 23, 2015 at 3:08
  • $\begingroup$ Quadratic Beta term in order to find it's coefficient, instead of a quadratic X term (but please correct me if I'm mistaken). $\endgroup$
    – John Madden
    Commented Sep 23, 2015 at 3:08
  • 1
    $\begingroup$ Sorry, I just realized I muddled up my comment earlier; please ignore it (in fact I'll remove it to avoid confusing anyone else). Of course $\beta$ is the variable here. The quadratic term is in the form $\beta^\top V^{-1} \beta$ (for some $V$); the two linear terms are in $\beta$ and $\beta^\top$. Those terms are scalars and so can be transposed and combined (NB $\beta$ is not a scalar, the term involving $\beta$ is). A more general version of your question is solved here $\endgroup$
    – Glen_b
    Commented Sep 23, 2015 at 6:31
  • $\begingroup$ Glen, I don't know how I didn't see this earlier, but this is the correct solution. Thank you for your help, I ended up figuring it out eventually! $\endgroup$
    – John Madden
    Commented Nov 10, 2015 at 21:55
  • 1
    $\begingroup$ Good idea, will do $\endgroup$
    – John Madden
    Commented Nov 10, 2015 at 22:23

1 Answer 1

Reset to default
3
$\begingroup$

I'm posting what I now know to be the answer in case anyone ever stumbles upon this.

Notice that the -2 linear terms above are transposes of one another. Further, notice that the -2 linear terms above have dimension 1X1. Consider that the transpose of a scalar is itself. Therefore, we can combine the two terms.

$\beta'X'X\beta-\beta'X'y -\beta'X'y = \beta'X'X\beta-2\beta'X'y$

As I stated in my question, the coefficient of the quadratic term is the inverse of the variance, and the coefficient of the -2 linear term is the mean/variance.

So, variance = $(X'X)^{-1}$

mean = $(X'X)^{-1}X'y$ (We know to multiply on the left because it only makes sense one way dimensionally), as desired.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.