# Bayesian inference of mean of Multivariate Gaussian

Text of the problem I have to solve..

State the generic case of a Bayesian treatment of the parameters $\theta$ used to model the data, i.e., $p( \theta | X, Y)$. Derive the estimate of the mean, assuming that the model of the output variable is a Gaussian distribution with a fixed variance. Which are the advantages of being Bayesian?

My (incomplete) answer:

As always we use the maximul likelihood approach:

\begin{equation} \mathrm{arg}\max_{ \theta} \log P( \theta | X, Y) = \mathrm{arg}\max_{ \theta} \log P( Y | X, \theta)P( \theta). \end{equation}

We could assume $P( \theta)$ normal distributed with $0$ mean and $\Sigma_{ \theta}$ as standard deviation. As we assume the data i.i.d, we could derive:

\begin{equation} \mathrm{arg} \max_{ \theta} \sum_i \log P(y_i | X, \theta) + \log P( \theta). \end{equation}

Now, since we also have assumed the output distributed as Gaussian, we will end up in the following:

\begin{equation} \mathrm{arg}\max_{ \mu} \sum_i - \frac{1}{2} \log(2^d \pi^d |\Sigma|) - \frac{1}{2} (X-\mu)^T \Sigma^{-1} (X- \mu) - \frac{1}{2} \log(2^d \pi^d |\Sigma_{ \mu}|) - \frac{1}{2} (\mu)^T \Sigma^{-1} (\mu) \end{equation}

where $d$ is the number of features. Since the function is convex, we could use the gradient:

\begin{equation} \nabla{ \mu} \sum_i - \frac{1}{2} \log(2^d \pi |\Sigma|) - \frac{1}{2} (y_i-\mu)\Sigma (y_i- \mu)^T - \frac{1}{2} \log(2^d \pi |\Sigma_{ \mu}|) - \frac{1}{2} (\mu)\Sigma (\mu)^T = 0 \end{equation} thus: \begin{equation} \nabla_{ \mu} \sum_i - \frac{1}{2} (y_i-\mu)^T\Sigma^{-1} (y_i- \mu) - \frac{1}{2} (\mu)^T\Sigma_{ \mu}^{-1} (\mu) = 0. \end{equation} We already know how to compute $\nabla_{ \mu} (y_i-\mu)^T \Sigma^{-1} (y_i- \mu)$ and $( \mu)\Sigma_{ \mu} ( \mu)^T$:

\begin{equation} \sum_i (y_i-\mu)^T (\Sigma^{-1}+ \Sigma^{-T}) + (\mu)^T (\Sigma_{ \mu}^{-1} + \Sigma_{ \mu}^{-T}) = 0, \end{equation}

considering $\Sigma$ and $\Sigma_{\mu}$ be symmetric then

\begin{equation} \sum_i (y_i-\mu)^T \Sigma^{-1} + (\mu)^T \Sigma_{ \mu}^{-1} = 0 \end{equation}

which yelds:

\begin{equation} - N \mu^T \Sigma^{-1} + N (\mu)^T \Sigma_{ \mu}^{-1} + \sum_i y_i^T \Sigma^{-1} = 0. \end{equation}

We must isolate $\mu^T$

\begin{equation} + N \mu^T - N (\mu)^T \Sigma_{ \mu}^{-1} \Sigma = \sum_i y_i^T \end{equation}

\begin{equation} (\mu)^T (I- \Sigma_{ \mu}^{-1} \Sigma ) = \frac{\sum_i y_i^T}{N} \end{equation}

From here I can't go further. I don't even know if till here it is correct what I did.

## 1 Answer

Hints:

1. The question is too vague as stated, in particular the difference between $X$ and $Y$ is not spelled out. There is no $y_i$ in your developed equations.
2. It is unclear why you follow a maximisation path when the question does not mention MAP (maximum a posteriori). A more natural approach would be to seek the posterior mean (provided it is well-defined).
3. Seeking directly the posterior distribution on $\mu$ is actually simpler. This is how the problem is processed in Bayesian textbooks (like ours).
4. What is the meaning of $-$ in $\Sigma_{\mu}^{-T}$?
• 1) I assume $Y$ i.i.d thus $P(Y | X, \theta) = \prod_i P(y_i | X, \theta)$. 2) could you provide an example? 3) - same as 2) . 4) $\Sigma_{\mu}^{-T} = (\Sigma_{\mu}^{T})^{-1}$. P.S. I found many typos, now I'll correct them. – Sam Jun 8 '17 at 9:55
• 2-3) check textbooks. 4) $\Sigma_\mu$ is a symmetric matrix, you do not need the transpose sign. There are still typos in the formulas, like confusion between the different covariance matrices. – Xi'an Jun 8 '17 at 10:04