3
$\begingroup$

I'm working on an online method to adapt the parameters $\mu, \Sigma$ of a Gaussian distribution. Do to so i perform a gradient descent on the log likelihood $L$. With the help of the matrix cookbook i've found that the partial derivatives $\nabla_\Sigma L$ and $\nabla_\mu L$ are given by

\begin{align} \nabla_\Sigma L = \frac{ \partial {\bf L} }{ \partial {\boldsymbol \Sigma}} &= -\frac{1}{2} \left( {\boldsymbol \Sigma}^{-1} - {\boldsymbol \Sigma}^{-1} \left( {\boldsymbol y} - {\boldsymbol \mu} \right) \left( {\boldsymbol y} - {\boldsymbol \mu} \right)' {\boldsymbol \Sigma}^{-1} \right) \end{align} and \begin{align} \nabla_\mu L = \frac{ \partial {\bf L} }{ \partial {\boldsymbol \mu}} &= {\boldsymbol \Sigma}^{-1} \left( {\boldsymbol y} - {\boldsymbol \mu} \right) \end{align} where $\boldsymbol{y}$ are the training samples and $\boldsymbol{L}$ the log likelihood of the multivariate gaussian distribution given by $\mu$ and $\Sigma$. I'm setting a learning rate $\alpha$ and proceed in the following way:

  1. Sample an $y$ from unknown $p_\theta(y)$.
  2. Compute gradients $\nabla_\Sigma L$ and $\nabla_\mu L$
  3. Update: $\mu = \mu + \nabla_\mu L \alpha$, and $\Sigma = \Sigma + \nabla_\Sigma L \alpha$

I've set $\alpha = 0.002$ and $p_\theta(y) = \mathcal{N}(y | 0, 1)$, but it is not convering at all. What am i missing?

P.S.: I know there are different ways, but i'm trying to this using a gradient ascent methood.

Improve this question
$\endgroup$
5
  • 4
    $\begingroup$ Could you please explain your second equation in pt number 3? $\endgroup$
    – Sid
    Commented Apr 9, 2018 at 22:08
  • $\begingroup$ Sorry, that was a mistake. I fixed it. $\endgroup$
    – hh32
    Commented Apr 10, 2018 at 5:07
  • $\begingroup$ Just to make sure the error is nothing overly simple: For the univariate case we have $N(y|0,1) = ce^{-(y-\mu)^2/2\Sigma}$ so that $-\log \text{Likelihood} = \text{const} + -(-(y-\mu)^2/2\Sigma) = \text{const} + (y-\mu)^2/2\Sigma$ so that the derivative is $\partial_\mu -\log L = \Sigma^{-1} (y-\mu)$. You want to maximize the likelihood, i.e. maximize $\log L$ i.e. minimize $-\log L$. The gradient points into the direction of the steepest increase in function values, i.e. shouldn't you subtract the gradient in order to minimize? $\endgroup$
    – Fabian Werner
    Commented Apr 10, 2018 at 7:10
  • $\begingroup$ @FabianWerner what you wrote is correct, but I'm taking the derivative of $\log L$, not $-\log L$ so i add the gradient. It should be same. $\endgroup$
    – hh32
    Commented Apr 10, 2018 at 9:04
  • $\begingroup$ @hh32: that is exactly my point: the partial of -log L (w.r.t. $\mu$) and the one of log L coincide. One of us must have made a mistake... I think you are taking the gradient of -log L, hence you need to minimize, hence subtract... or was it me who committed the error? $\endgroup$
    – Fabian Werner
    Commented Apr 10, 2018 at 11:58

1 Answer 1

Reset to default
1
$\begingroup$

One possibility is that the sample covariance matrix is becoming near singular somewhere along the way, and your iterations involve inverting this matrix

The eigenvalues of your sample covariance matrix will follow a Marcenko-Pastur distribution (https://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution), and from this distribution you will see that it is not inconceivable that you get high condition numbers (the condition number is the ratio of the largest to the smallest eigenvalue of your covariance matrix)

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.