I have been trying to derive the posterior distribution in the case of weighted Bayesian regression in the case of multivariate normal distribution for a few days and have been stuck. I am not sure if this is due to lack of my linear algebra skills or if I have done something really wrong.

In my case, the distributional assumptions are:

$$ y = N (B x, W^{-1} \nabla) $$

The parameters $B$ have a normal prior.

$$ P(B) = N (B_0, \Sigma_{B}) $$

The conditional posterior distribution for $B$ is proportional to $P(y|B) P(B)$. This is proportional to:

$$ \exp \big((B - B_0)^T \Sigma (B - B_0) + (y - Bx)^T \nabla (y - Bx)\big) $$

I have seen the textbooks mention something called completing the square which as far as I can tell equates to matching like terms. Even though I could work out for scalar variance terms, I am having trouble figuring out how to deal with the precision matrices and how this can be decoupled.


So, I tried to make a run with Glen_b's suggestion and came this far:

Original expression without the exponential (I will simplify notation slightly):

$$ (B - B_0)^T \Sigma (B - B_0) + ( y - Bx)^T \nabla (y - Bx) $$ Expanding:

$$ B^T\Sigma B - 2 B_0^T \Sigma B + B_0^T \Sigma B_0 + y^T \nabla y - 2 y^T \nabla B x + (Bx)^T \nabla Bx $$

$$ = B^T(\Sigma + x^T \nabla x)B - 2 (B_0^T \Sigma B + y^T \nabla B x) + C $$

So I am not sure how to make further progress with the second term. I guess I have to make it in the form 2(some_term)B but not sure how that can be done.

  • 2
    $\begingroup$ Good resource for steps to completing the square for a multivariate (or univariate) Gaussian likelihood and prior: learnbayes.org/… $\endgroup$
    – JStrahl
    Aug 30 '18 at 11:23

The essential step is along these lines:

$$(x-a)^TA(x-a) + (x-b)^TB(x-b)$$

$$=x^TAx -2a^TAx + a^TAa+ x^TBx -2b^TBx + b^TBb$$

$$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$

$$=x^T(A+B)x -2(a^TA+b^TB)x + (a^TAa+ b^TBb)$$

Let $M=A+B$, let $m=(M^{-1})^T(B^Tb+A^Ta)$ and let $C=a^TAa+ b^TBb-m^TMm$

(I hope I got $m$ right there. You just need to pick $m$ so that $a^TA+b^TB=m^TM$ given the definition of $M$ we had already. And feel free to work with $-2x^T(Aa+Bb)$ if that's easier.)



Taking your expression, putting it all in one exponent, then focusing on what's in the exponent, and dropping off constants not in $\beta$ (like the factor of $\frac{1}{2}$, we get something of the above form).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.