# Computing the conditional distribution for the mean of a Gaussian

I have the following distributional assumptions on some on my RV and model parameters:

$$y_i \sim N(\beta x_i, w_i^{-1}\Sigma_y)$$

There is a normal prior on the parameters $\beta$ as well:

$$\beta \sim N(\beta_0, \Sigma_{\beta})$$

AFAIK, the conditional distribution for the mean parameter $\beta$, given by $P(\beta|y, w) should also be normally distributed due to conjugate relationship. I have been struggling to derive that. So, I want to get the expression for the conditional distribution of$\beta$, which in this case is given by$p(y|\beta, w) P(\beta)$and this is proportional to: $$|\Sigma_y|^\frac{-N}{2}\Bigg(\exp \sum_{i=1}^N(y_i - \beta x_i)^T\Sigma_y^{-1}(y-\beta x_i) \Bigg)|\Sigma_{\beta}|^{-0.5} \exp (\beta-\beta_0)^T\Sigma_{\beta}^{-1}(\beta -\beta_0)$$ Here I am dropping the terms depending on$w$. This becomes $$|\Sigma_y|^\frac{-N}{2} |\Sigma_{\beta}|^{-0.5} \exp \big((\beta-\beta_0)^T\Sigma_{\beta}^{-1}(\beta -\beta_0) + \sum_{i=1}^N(y_i - \beta x_i)^T\Sigma_y^{-1}(y-\beta x_i)\big)$$ However, after this I am quite stuck. I am not sure how I can express this in the form of a normal distribution. ## 1 Answer First note that the availability of a closed-form posterior, which essentially amounts to conjugacy as you notice, depends also on$\sigma^2$and on what prior distribution it is assigned. If$\sigma^2$is known, then your prior$\beta \sim N(\beta_0, \Sigma_0)$is indeed conjugate, otherwise if$\sigma^2$is random, you need an inverse gamma prior on it, and you also need to change your prior on$\beta$as $$\beta\vert\sigma^2 \sim N(\beta_0, \sigma^2\Sigma_0).$$ Showing conjugacy requires some tedious algebra, a crucial point in the algebra is to use the least squares estimate $$\hat \beta = (XX^T)^{-1}X^Ty$$ in order to center the posterior distribution of$\beta$. You can show that the posterior is $$\beta\vert \sigma^2, Y, X\sim N(\beta_n, \sigma^2\Sigma_n)$$ where $$\Sigma_n = (X^TX+\Sigma_0^{-1})^{-1},\quad \beta_n = \Sigma_n((X^TX)\hat\beta+\Sigma_0^{-1}\beta_0).$$ The steps are well detailed on the Wikipedia page of Bayesian linear regression, see here: http://en.wikipedia.org/wiki/Bayesian_linear_regression#Posterior_distribution PS: mind your expression of the posterior distribution for$\beta$where you are missing some terms. It should read $$\Bigg(\prod_{i=1}^N \exp -\frac{w_i}{2 \sigma^2}\big(y_i - \beta^Tx_i\big)^T\big(y_i - \beta^Tx_i\big)\Bigg)\exp -\frac{1}{2 \sigma^2}(\beta-\beta_0)^T\Sigma_0^{-1}(\beta -\beta_0)$$ • Thanks for your reply. May I ask what this$\sigma^2$term is. I do not see it. Is the same as the variace weights$w_i$in my case? In my case, the variance for$y_i$is given by$w_i^{-1} \Sigma_y$as$y$is a vector quantity. – Luca Feb 26, 2015 at 15:57 • the$w_i$is like a concentration parameter while the inverse$\sigma^2$is the variance. I was meaning the parameter that you had in the model equation in the previous version of your question. Feb 26, 2015 at 16:02 • Ah ok. Sorry, I reformulated it to use a multivariate form. So, if I understand you correctly, I need to change the prior on$\beta$as$(\beta_0, \Sigma_{y} \Sigma_{\beta})\$. I was wondering if you would be kind enough to elaborate on why this form is necessary?
– Luca
Feb 26, 2015 at 16:06