Gradient of multivariate Gaussian log-likelihood I'm trying to find the MAP estimate for a model by gradient descent.  My prior is multivariate Gaussian with a known covariance matrix.
On a conceptual level, I think I know how to do this, but I was hoping for some help with the details.  In particular, if there is an easier way to approach the problem, then that would be especially useful.
Here's what I think I need to do:


*

*For each dimension, find the conditional distribution, given my current position in the other dimensions.

*This gives me a local univariate Gaussian in each dimension, with the correct mean and standard deviation.

*I think that the gradient should just be a vector of derivatives for each of these univariate distributions.


My question has two parts:


*

*Is this the best approach to take, or is there an easier way?

*Assuming I need to go this route, what's the best way to go about finding these conditional distributions?

 A: What about optimization?
Let's see if I understand you correctly. You have a model $p(y|x, \theta)$ conditioned on some observation $x$ and a set of parameters $\theta$ and a prior $p(\theta)$ leading to a joint likelihood of $\mathcal{L} = p(y|x, \theta)p(\theta)$. The parameters are distributed according to a known multivariate normal, i.e. $\theta \sim \mathcal{N}(\mu, \Sigma)$. You want to find the MAP solution to this problem, i.e.
$$
\text{argmax}_{\theta} \mathcal{L}.
$$
A special case of this problem is well studied in the neural networks community, known as weight decay. In that case, $\mu=\mathbf{0}$ and $\Sigma = \mathbf{I}\sigma^2$.
As you already noted, the trick is that $\text{argmax}_{\theta} \mathcal{L} = \text{argmax}_{\theta} \log \mathcal{L}$. When you take the log of the Gaussian density, many ugly terms (the exponential) vanish and you will end up with sth like $\log p(\theta) = -{1 \over 2}(\theta - \mu)^T\Sigma^{-1}(\theta - \mu) + \text{const}$. If you differentiate that, Sam Roweis' matrix identities will come in handy and let you arrive at
$$
-{1 \over 2}{\partial (\theta - \mu)^T\Sigma^{-1}(\theta - \mu) \over \partial \theta} = -\Sigma^{-1}(\theta - \mu).
$$
(Please verify, this was done quickly and in my head.) Together with the derivatives of your model, you can use off-the-shelf optimizers to arrive at a MAP solution.
Update: Incorporated comment by David J. Harris. Formulas should be correct now.
A: If the likelihood is not Gaussian it's not possible to say if there are analytic results. Also, the second bullet is then incorrect in general. Since Gaussian prior and general likelihood does not make for conditional gaussian distributions on the vector components. 
One way to get the MAP would be to do a full Bayesian analysis, e.g. using MCMC and use the samples from the posterior to estimate it. [In which case you'd have better information available than only using the MAP.] Out of interest- why not go down this route anyhow?
Another approach could be to do (I've not seen this done generally so someone please correct me if it's nuts):
$ p(\theta|x) = \frac{p(x|\theta)p(\theta)}{p(x)}$
$ l(\theta|x) = l(x|\theta) + l(\theta) - l(x) $
$ \frac{dl(\theta|x)}{d\theta} = \frac{dl(x|\theta)}{d\theta} + \frac{dl(\theta)}{d\theta} = 0$
Then solve for $\theta$ (probably numerically).
