Beginner here, apologies if this is something very simple. I am trying to do a gradient ascent to estimate means for a Mixture of Gaussian model. I am using $(x-µ)/σ^3(2π)^{(1/2)} * e^{-((x-µ)/σ)^2)/2}$ for the first derivative of the gaussian distribution. My question here is, for the gradient ascent should I sum the first order derivative over all values of $x$ (datapoints) for each update?

  • $\begingroup$ Do you assume the other parameters are known: variances and mixture weights? $\endgroup$ – Benoit Sanchez Dec 3 '17 at 19:14
  • $\begingroup$ Yes, I have the other parameters. But not the means. Further, with gradient ascent, if I sum over all datapoints, it gets stuck at a local maxima while EM algo. doesn't in fact EM converges thrice faster than gradient. They both start at same initial mean values $\endgroup$ – Guest Dec 3 '17 at 23:28

What most people do is work on the $\log$ of the likelihood. The likelihood is multiplicative across data points, thus the log-likelihood is additive. Maximizing one or the other is the same. Formally, call $L_i$ the likelihood of point number $i$.

$$\log(L)=\sum_i \log(L_i)$$

For a mixture of $d$ Gaussians:

$$\log(L_i)=\log\left(\sum_{j=1}^d \lambda_j\frac{1}{\sqrt{2\pi\sigma_j^2}}e^{-(x_i-\mu_j)^2/2\sigma_j^2}\right)$$

So first, take the derivative of $\log(L_i)$ for each unknown parameter: each mean $\mu_j$ (you assume variance $\sigma_j^2$ and weight $\lambda_j$ are known). Then sum it over all data points to get the global gradient.

Note that Gaussian mixture is usually not solved with gradient ascent but Expectation/Maximization since the log likelihood is not convex (at least with all parameters unknown, I don't known if only the means are unknown). Yet I've seen non convex functions successfully maximized by gradient ascent, so it may be interesting to try.

| cite | improve this answer | |
  • $\begingroup$ The issue here is, given the huge dataset that I am working with, iterating over all the elements for each update would be computationally prohibitive. $\endgroup$ – Guest Dec 4 '17 at 20:08
  • $\begingroup$ The classical way to accelerate gradient decent is stochastic gradient descent. It's especially efficient on big datasets. It's not more difficult to implement, maybe even simpler. $\endgroup$ – Benoit Sanchez Dec 5 '17 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.