# Gradient ascent to maximise log likelihood

I'm working on an online method to adapt the parameters $\mu, \Sigma$ of a Gaussian distribution. Do to so i perform a gradient descent on the log likelihood $L$. With the help of the matrix cookbook i've found that the partial derivatives $\nabla_\Sigma L$ and $\nabla_\mu L$ are given by

\begin{align} \nabla_\Sigma L = \frac{ \partial {\bf L} }{ \partial {\boldsymbol \Sigma}} &= -\frac{1}{2} \left( {\boldsymbol \Sigma}^{-1} - {\boldsymbol \Sigma}^{-1} \left( {\boldsymbol y} - {\boldsymbol \mu} \right) \left( {\boldsymbol y} - {\boldsymbol \mu} \right)' {\boldsymbol \Sigma}^{-1} \right) \end{align} and \begin{align} \nabla_\mu L = \frac{ \partial {\bf L} }{ \partial {\boldsymbol \mu}} &= {\boldsymbol \Sigma}^{-1} \left( {\boldsymbol y} - {\boldsymbol \mu} \right) \end{align} where $\boldsymbol{y}$ are the training samples and $\boldsymbol{L}$ the log likelihood of the multivariate gaussian distribution given by $\mu$ and $\Sigma$. I'm setting a learning rate $\alpha$ and proceed in the following way:

1. Sample an $y$ from unknown $p_\theta(y)$.
2. Compute gradients $\nabla_\Sigma L$ and $\nabla_\mu L$
3. Update: $\mu = \mu + \nabla_\mu L \alpha$, and $\Sigma = \Sigma + \nabla_\Sigma L \alpha$

I've set $\alpha = 0.002$ and $p_\theta(y) = \mathcal{N}(y | 0, 1)$, but it is not convering at all. What am i missing?

P.S.: I know there are different ways, but i'm trying to this using a gradient ascent methood.

• Could you please explain your second equation in pt number 3? – Sid Apr 9 '18 at 22:08
• Sorry, that was a mistake. I fixed it. – hh32 Apr 10 '18 at 5:07
• Just to make sure the error is nothing overly simple: For the univariate case we have $N(y|0,1) = ce^{-(y-\mu)^2/2\Sigma}$ so that $-\log \text{Likelihood} = \text{const} + -(-(y-\mu)^2/2\Sigma) = \text{const} + (y-\mu)^2/2\Sigma$ so that the derivative is $\partial_\mu -\log L = \Sigma^{-1} (y-\mu)$. You want to maximize the likelihood, i.e. maximize $\log L$ i.e. minimize $-\log L$. The gradient points into the direction of the steepest increase in function values, i.e. shouldn't you subtract the gradient in order to minimize? – Fabian Werner Apr 10 '18 at 7:10
• @FabianWerner what you wrote is correct, but I'm taking the derivative of $\log L$, not $-\log L$ so i add the gradient. It should be same. – hh32 Apr 10 '18 at 9:04
• @hh32: that is exactly my point: the partial of -log L (w.r.t. $\mu$) and the one of log L coincide. One of us must have made a mistake... I think you are taking the gradient of -log L, hence you need to minimize, hence subtract... or was it me who committed the error? – Fabian Werner Apr 10 '18 at 11:58