I'm building a mixture model consists 2 normal distribution:

\begin{equation} X_1 \sim \mathcal{N}(\mu_1,\sigma_{1}^{2}), \quad X_2 \sim \mathcal{N}(\mu_2,\sigma_{2}^{2})\end{equation}


$$pdf(x) = p_1 N(x, \mu_1, \sigma_1^2) + p_2 N(x, \mu_2, \sigma_2^2)$$

where $p_1+p_2=1$,

and $N(x,\mu,\sigma) = \frac{1}{\sqrt{2\pi \sigma^2}}exp(-\frac{(x-\mu)^2}{2\sigma^2})$.

With samples ready, I'm beginning to fit $p_1$, $\mu_1$, $\sigma_1$ and $p_2$, $\mu_2$, $\sigma_2$.

First, I defined a very simple fitting way, by frequency count.

  1. I split the range of $x$ to different bands, each band's width $w=\sigma/10$, where $\sigma$ is the standard deviation of the samples.
  2. I count the frequency of $x$ in each band $Freq_i = \text{Sample Count in band_i} / \text{Total Sample Count}$
  3. I defined the target as $Target_{FreqFitting} = \Sigma_i{(Freq_{i, sample} - Freq_{i, model})^2}$

Using excel "solver" to solve the problem, it looks nice: enter image description here

Here the black line is the samples' frequency; solid blue line is a normal distribution model sharing the same $\mu$ and $\sigma$ of the samples, it's not close to the real samples; red line is the result of Gaussian mixture model, while the dotted lines are the two components.

Then, I went to Maximum Likelyhood Estimation. It turns out a surprise.

The target is $Target_{MLE} = \Sigma_i{Ln(pdf(x_i))}$. The result is quite disappointing:enter image description here

The green line is the result of MLE fitting, it's way too bad.

After checking the details, it seems that MLE optimize for samples that are far away from the mean - the target gain from $x$ beyond $[\mu-3\sigma, \mu+3\sigma]$.

Thirdly, I manipulated the MLE target functions a bit: $Target_{AdjustedMLE} = \Sigma_i{\frac{Ln(pdf(x_i))}{1+a((x-\mu)/\sigma)^2} }$

If I choose $a=0.5$, it fits well: enter image description here

Is this a weakness of MLE?

Do you guys meet similiar problem using MLE? how do you achieve a good fitting?


No answer yet too, but how do you account for the constrain p1 + p2 = 1 in your max likelihood estimate ?

I would replace $p_2$ by $1 - p_1$ in your model and fit only $\mu_1$, $\sigma_1$, $\mu_2$, $\sigma_2$ and $p_1$.

  • $\begingroup$ yeah $p_2=1-p_1$, and also $p_1 \mu_1 + p_2 \mu_2 = \mu$. in excel i just put the formula and let the solver try. $\endgroup$ – athos Sep 18 '13 at 14:46

no answer yet. Just to say i put a threshold of the pdf, which i chose to be pdf of normal distribution at 6 sigma, then anything below the threshold, will be assigned as the threshold's issue. in this way i managed to suppress MLE's focus on outliers.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.