# Maximum likelihood estimation get biased by fat-tail samples identifying mixture model of 2 normal distributions

I'm building a mixture model consists 2 normal distribution:

$$X_1 \sim \mathcal{N}(\mu_1,\sigma_{1}^{2}), \quad X_2 \sim \mathcal{N}(\mu_2,\sigma_{2}^{2})$$

while

$$pdf(x) = p_1 N(x, \mu_1, \sigma_1^2) + p_2 N(x, \mu_2, \sigma_2^2)$$

where $p_1+p_2=1$,

and $N(x,\mu,\sigma) = \frac{1}{\sqrt{2\pi \sigma^2}}exp(-\frac{(x-\mu)^2}{2\sigma^2})$.

With samples ready, I'm beginning to fit $p_1$, $\mu_1$, $\sigma_1$ and $p_2$, $\mu_2$, $\sigma_2$.

First, I defined a very simple fitting way, by frequency count.

1. I split the range of $x$ to different bands, each band's width $w=\sigma/10$, where $\sigma$ is the standard deviation of the samples.
2. I count the frequency of $x$ in each band $Freq_i = \text{Sample Count in band_i} / \text{Total Sample Count}$
3. I defined the target as $Target_{FreqFitting} = \Sigma_i{(Freq_{i, sample} - Freq_{i, model})^2}$

Using excel "solver" to solve the problem, it looks nice:

Here the black line is the samples' frequency; solid blue line is a normal distribution model sharing the same $\mu$ and $\sigma$ of the samples, it's not close to the real samples; red line is the result of Gaussian mixture model, while the dotted lines are the two components.

Then, I went to Maximum Likelyhood Estimation. It turns out a surprise.

The target is $Target_{MLE} = \Sigma_i{Ln(pdf(x_i))}$. The result is quite disappointing:

The green line is the result of MLE fitting, it's way too bad.

After checking the details, it seems that MLE optimize for samples that are far away from the mean - the target gain from $x$ beyond $[\mu-3\sigma, \mu+3\sigma]$.

Thirdly, I manipulated the MLE target functions a bit: $Target_{AdjustedMLE} = \Sigma_i{\frac{Ln(pdf(x_i))}{1+a((x-\mu)/\sigma)^2} }$

If I choose $a=0.5$, it fits well:

Is this a weakness of MLE?

Do you guys meet similiar problem using MLE? how do you achieve a good fitting?

I would replace $p_2$ by $1 - p_1$ in your model and fit only $\mu_1$, $\sigma_1$, $\mu_2$, $\sigma_2$ and $p_1$.
• yeah $p_2=1-p_1$, and also $p_1 \mu_1 + p_2 \mu_2 = \mu$. in excel i just put the formula and let the solver try. – athos Sep 18 '13 at 14:46