3
$\begingroup$

I have the following distributional assumptions on some on my RV and model parameters:

$$ y_i \sim N(\beta x_i, w_i^{-1}\Sigma_y) $$

There is a normal prior on the parameters $\beta$ as well:

$$ \beta \sim N(\beta_0, \Sigma_{\beta}) $$

AFAIK, the conditional distribution for the mean parameter $\beta$, given by $P(\beta|y, w) should also be normally distributed due to conjugate relationship. I have been struggling to derive that.

So, I want to get the expression for the conditional distribution of $\beta$, which in this case is given by $p(y|\beta, w) P(\beta)$ and this is proportional to:

$$ |\Sigma_y|^\frac{-N}{2}\Bigg(\exp \sum_{i=1}^N(y_i - \beta x_i)^T\Sigma_y^{-1}(y-\beta x_i) \Bigg)|\Sigma_{\beta}|^{-0.5} \exp (\beta-\beta_0)^T\Sigma_{\beta}^{-1}(\beta -\beta_0) $$

Here I am dropping the terms depending on $w$. This becomes $$ |\Sigma_y|^\frac{-N}{2} |\Sigma_{\beta}|^{-0.5} \exp \big((\beta-\beta_0)^T\Sigma_{\beta}^{-1}(\beta -\beta_0) + \sum_{i=1}^N(y_i - \beta x_i)^T\Sigma_y^{-1}(y-\beta x_i)\big) $$

However, after this I am quite stuck. I am not sure how I can express this in the form of a normal distribution.

$\endgroup$
1
$\begingroup$

First note that the availability of a closed-form posterior, which essentially amounts to conjugacy as you notice, depends also on $\sigma^2$ and on what prior distribution it is assigned. If $\sigma^2$ is known, then your prior $\beta \sim N(\beta_0, \Sigma_0)$ is indeed conjugate, otherwise if $\sigma^2$ is random, you need an inverse gamma prior on it, and you also need to change your prior on $\beta$ as $$\beta\vert\sigma^2 \sim N(\beta_0, \sigma^2\Sigma_0).$$

Showing conjugacy requires some tedious algebra, a crucial point in the algebra is to use the least squares estimate $$\hat \beta = (XX^T)^{-1}X^Ty$$ in order to center the posterior distribution of $\beta$. You can show that the posterior is $$\beta\vert \sigma^2, Y, X\sim N(\beta_n, \sigma^2\Sigma_n)$$ where $$\Sigma_n = (X^TX+\Sigma_0^{-1})^{-1},\quad \beta_n = \Sigma_n((X^TX)\hat\beta+\Sigma_0^{-1}\beta_0).$$

The steps are well detailed on the Wikipedia page of Bayesian linear regression, see here: http://en.wikipedia.org/wiki/Bayesian_linear_regression#Posterior_distribution

PS: mind your expression of the posterior distribution for $\beta$ where you are missing some terms. It should read $$\Bigg(\prod_{i=1}^N \exp -\frac{w_i}{2 \sigma^2}\big(y_i - \beta^Tx_i\big)^T\big(y_i - \beta^Tx_i\big)\Bigg)\exp -\frac{1}{2 \sigma^2}(\beta-\beta_0)^T\Sigma_0^{-1}(\beta -\beta_0)$$

$\endgroup$
  • $\begingroup$ Thanks for your reply. May I ask what this $\sigma^2$ term is. I do not see it. Is the same as the variace weights $w_i$ in my case? In my case, the variance for $y_i$ is given by $w_i^{-1} \Sigma_y$ as $y$ is a vector quantity. $\endgroup$ – Luca Feb 26 '15 at 15:57
  • 1
    $\begingroup$ the $w_i$ is like a concentration parameter while the inverse $\sigma^2$ is the variance. I was meaning the parameter that you had in the model equation in the previous version of your question. $\endgroup$ – julyan Feb 26 '15 at 16:02
  • $\begingroup$ Ah ok. Sorry, I reformulated it to use a multivariate form. So, if I understand you correctly, I need to change the prior on $\beta$ as $(\beta_0, \Sigma_{y} \Sigma_{\beta})$. I was wondering if you would be kind enough to elaborate on why this form is necessary? $\endgroup$ – Luca Feb 26 '15 at 16:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.