# Gradient Ascent for Mixture of Gaussians

Beginner here, apologies if this is something very simple. I am trying to do a gradient ascent to estimate means for a Mixture of Gaussian model. I am using $(x-µ)/σ^3(2π)^{(1/2)} * e^{-((x-µ)/σ)^2)/2}$ for the first derivative of the gaussian distribution. My question here is, for the gradient ascent should I sum the first order derivative over all values of $x$ (datapoints) for each update?

• Do you assume the other parameters are known: variances and mixture weights? – Benoit Sanchez Dec 3 '17 at 19:14
• Yes, I have the other parameters. But not the means. Further, with gradient ascent, if I sum over all datapoints, it gets stuck at a local maxima while EM algo. doesn't in fact EM converges thrice faster than gradient. They both start at same initial mean values – Guest Dec 3 '17 at 23:28

What most people do is work on the $\log$ of the likelihood. The likelihood is multiplicative across data points, thus the log-likelihood is additive. Maximizing one or the other is the same. Formally, call $L_i$ the likelihood of point number $i$.

$$\log(L)=\sum_i \log(L_i)$$

For a mixture of $d$ Gaussians:

$$\log(L_i)=\log\left(\sum_{j=1}^d \lambda_j\frac{1}{\sqrt{2\pi\sigma_j^2}}e^{-(x_i-\mu_j)^2/2\sigma_j^2}\right)$$

So first, take the derivative of $\log(L_i)$ for each unknown parameter: each mean $\mu_j$ (you assume variance $\sigma_j^2$ and weight $\lambda_j$ are known). Then sum it over all data points to get the global gradient.

Note that Gaussian mixture is usually not solved with gradient ascent but Expectation/Maximization since the log likelihood is not convex (at least with all parameters unknown, I don't known if only the means are unknown). Yet I've seen non convex functions successfully maximized by gradient ascent, so it may be interesting to try.

• The issue here is, given the huge dataset that I am working with, iterating over all the elements for each update would be computationally prohibitive. – Guest Dec 4 '17 at 20:08
• The classical way to accelerate gradient decent is stochastic gradient descent. It's especially efficient on big datasets. It's not more difficult to implement, maybe even simpler. – Benoit Sanchez Dec 5 '17 at 11:05