Text of the problem I have to solve..

State the generic case of a Bayesian treatment of the parameters $\theta$ used to model the data, i.e., $p( \theta | X, Y)$. Derive the estimate of the mean, assuming that the model of the output variable is a Gaussian distribution with a fixed variance. Which are the advantages of being Bayesian?

My (incomplete) answer:

As always we use the maximul likelihood approach:

\begin{equation} \mathrm{arg}\max_{ \theta} \log P( \theta | X, Y) = \mathrm{arg}\max_{ \theta} \log P( Y | X, \theta)P( \theta). \end{equation}

We could assume $P( \theta)$ normal distributed with $0$ mean and $\Sigma_{ \theta}$ as standard deviation. As we assume the data i.i.d, we could derive:

\begin{equation} \mathrm{arg} \max_{ \theta} \sum_i \log P(y_i | X, \theta) + \log P( \theta). \end{equation}

Now, since we also have assumed the output distributed as Gaussian, we will end up in the following:

\begin{equation} \mathrm{arg}\max_{ \mu} \sum_i - \frac{1}{2} \log(2^d \pi^d |\Sigma|) - \frac{1}{2} (X-\mu)^T \Sigma^{-1} (X- \mu) - \frac{1}{2} \log(2^d \pi^d |\Sigma_{ \mu}|) - \frac{1}{2} (\mu)^T \Sigma^{-1} (\mu) \end{equation}

where $d$ is the number of features. Since the function is convex, we could use the gradient:

\begin{equation} \nabla{ \mu} \sum_i - \frac{1}{2} \log(2^d \pi |\Sigma|) - \frac{1}{2} (y_i-\mu)\Sigma (y_i- \mu)^T - \frac{1}{2} \log(2^d \pi |\Sigma_{ \mu}|) - \frac{1}{2} (\mu)\Sigma (\mu)^T = 0 \end{equation} thus: \begin{equation} \nabla_{ \mu} \sum_i - \frac{1}{2} (y_i-\mu)^T\Sigma^{-1} (y_i- \mu) - \frac{1}{2} (\mu)^T\Sigma_{ \mu}^{-1} (\mu) = 0. \end{equation} We already know how to compute $\nabla_{ \mu} (y_i-\mu)^T \Sigma^{-1} (y_i- \mu)$ and $( \mu)\Sigma_{ \mu} ( \mu)^T$:

\begin{equation} \sum_i (y_i-\mu)^T (\Sigma^{-1}+ \Sigma^{-T}) + (\mu)^T (\Sigma_{ \mu}^{-1} + \Sigma_{ \mu}^{-T}) = 0, \end{equation}

considering $\Sigma$ and $\Sigma_{\mu}$ be symmetric then

\begin{equation} \sum_i (y_i-\mu)^T \Sigma^{-1} + (\mu)^T \Sigma_{ \mu}^{-1} = 0 \end{equation}

which yelds:

\begin{equation} - N \mu^T \Sigma^{-1} + N (\mu)^T \Sigma_{ \mu}^{-1} + \sum_i y_i^T \Sigma^{-1} = 0. \end{equation}

We must isolate $\mu^T$

\begin{equation} + N \mu^T - N (\mu)^T \Sigma_{ \mu}^{-1} \Sigma = \sum_i y_i^T \end{equation}

\begin{equation} (\mu)^T (I- \Sigma_{ \mu}^{-1} \Sigma ) = \frac{\sum_i y_i^T}{N} \end{equation}

From here I can't go further. I don't even know if till here it is correct what I did.



  1. The question is too vague as stated, in particular the difference between $X$ and $Y$ is not spelled out. There is no $y_i$ in your developed equations.
  2. It is unclear why you follow a maximisation path when the question does not mention MAP (maximum a posteriori). A more natural approach would be to seek the posterior mean (provided it is well-defined).
  3. Seeking directly the posterior distribution on $\mu$ is actually simpler. This is how the problem is processed in Bayesian textbooks (like ours).
  4. What is the meaning of $-$ in $\Sigma_{\mu}^{-T}$?
  • 1
    $\begingroup$ 1) I assume $Y$ i.i.d thus $P(Y | X, \theta) = \prod_i P(y_i | X, \theta)$. 2) could you provide an example? 3) - same as 2) . 4) $\Sigma_{\mu}^{-T} = (\Sigma_{\mu}^{T})^{-1}$. P.S. I found many typos, now I'll correct them. $\endgroup$ – Sam Jun 8 '17 at 9:55
  • $\begingroup$ 2-3) check textbooks. 4) $\Sigma_\mu$ is a symmetric matrix, you do not need the transpose sign. There are still typos in the formulas, like confusion between the different covariance matrices. $\endgroup$ – Xi'an Jun 8 '17 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.