# Bayesian inference of mean of Multivariate Gaussian

Text of the problem I have to solve..

State the generic case of a Bayesian treatment of the parameters $\theta$ used to model the data, i.e., $p( \theta | X, Y)$. Derive the estimate of the mean, assuming that the model of the output variable is a Gaussian distribution with a fixed variance. Which are the advantages of being Bayesian?

As always we use the maximul likelihood approach:

$$\mathrm{arg}\max_{ \theta} \log P( \theta | X, Y) = \mathrm{arg}\max_{ \theta} \log P( Y | X, \theta)P( \theta).$$

We could assume $P( \theta)$ normal distributed with $0$ mean and $\Sigma_{ \theta}$ as standard deviation. As we assume the data i.i.d, we could derive:

$$\mathrm{arg} \max_{ \theta} \sum_i \log P(y_i | X, \theta) + \log P( \theta).$$

Now, since we also have assumed the output distributed as Gaussian, we will end up in the following:

$$\mathrm{arg}\max_{ \mu} \sum_i - \frac{1}{2} \log(2^d \pi^d |\Sigma|) - \frac{1}{2} (X-\mu)^T \Sigma^{-1} (X- \mu) - \frac{1}{2} \log(2^d \pi^d |\Sigma_{ \mu}|) - \frac{1}{2} (\mu)^T \Sigma^{-1} (\mu)$$

where $d$ is the number of features. Since the function is convex, we could use the gradient:

$$\nabla{ \mu} \sum_i - \frac{1}{2} \log(2^d \pi |\Sigma|) - \frac{1}{2} (y_i-\mu)\Sigma (y_i- \mu)^T - \frac{1}{2} \log(2^d \pi |\Sigma_{ \mu}|) - \frac{1}{2} (\mu)\Sigma (\mu)^T = 0$$ thus: $$\nabla_{ \mu} \sum_i - \frac{1}{2} (y_i-\mu)^T\Sigma^{-1} (y_i- \mu) - \frac{1}{2} (\mu)^T\Sigma_{ \mu}^{-1} (\mu) = 0.$$ We already know how to compute $\nabla_{ \mu} (y_i-\mu)^T \Sigma^{-1} (y_i- \mu)$ and $( \mu)\Sigma_{ \mu} ( \mu)^T$:

$$\sum_i (y_i-\mu)^T (\Sigma^{-1}+ \Sigma^{-T}) + (\mu)^T (\Sigma_{ \mu}^{-1} + \Sigma_{ \mu}^{-T}) = 0,$$

considering $\Sigma$ and $\Sigma_{\mu}$ be symmetric then

$$\sum_i (y_i-\mu)^T \Sigma^{-1} + (\mu)^T \Sigma_{ \mu}^{-1} = 0$$

which yelds:

$$- N \mu^T \Sigma^{-1} + N (\mu)^T \Sigma_{ \mu}^{-1} + \sum_i y_i^T \Sigma^{-1} = 0.$$

We must isolate $\mu^T$

$$+ N \mu^T - N (\mu)^T \Sigma_{ \mu}^{-1} \Sigma = \sum_i y_i^T$$

$$(\mu)^T (I- \Sigma_{ \mu}^{-1} \Sigma ) = \frac{\sum_i y_i^T}{N}$$

From here I can't go further. I don't even know if till here it is correct what I did.

1. The question is too vague as stated, in particular the difference between $X$ and $Y$ is not spelled out. There is no $y_i$ in your developed equations.
3. Seeking directly the posterior distribution on $\mu$ is actually simpler. This is how the problem is processed in Bayesian textbooks (like ours).
4. What is the meaning of $-$ in $\Sigma_{\mu}^{-T}$?
• 1) I assume $Y$ i.i.d thus $P(Y | X, \theta) = \prod_i P(y_i | X, \theta)$. 2) could you provide an example? 3) - same as 2) . 4) $\Sigma_{\mu}^{-T} = (\Sigma_{\mu}^{T})^{-1}$. P.S. I found many typos, now I'll correct them. – Sam Jun 8 '17 at 9:55
• 2-3) check textbooks. 4) $\Sigma_\mu$ is a symmetric matrix, you do not need the transpose sign. There are still typos in the formulas, like confusion between the different covariance matrices. – Xi'an Jun 8 '17 at 10:04