Why is maximum likelihood estimator suspectible to outliers?

Question

I'm new to statistics and currently learning abot MLE.

Some of the papers I read: Robust Graph Embedding with Noisy Link Weights mentioned MLEs are suspectible to contamination in data, but didn't make any further mention about it.

So I would like to know the reason for this with clear and detailed explains.

Answer 1

Measurements do not always show ideal behavior and the presumed underlying distribution that is used for the maximum likelihood estimate (MLE) is often not the distribution of the measurements.

For instance, in the graph above the distribution of the measurements is a mixture of two Gaussian distributions 25% with $\sigma = 10$ and 75% with $\sigma = 1$.

(so for whatever reason the distribution is not ideal, either it is because the population is not ideal or it is because the measurements are not perfect)

This component with large variation will increase the sampling variance (and inaccuracy/error) of the estimator a lot.

Instead of using the MLE (which in a simple case, like estimating the mean of the population, boils down to the average/mean value of the sample), one could use a statistic that filters a few of the extreme values from the sample. This would greatly reduce the variation of the statistic. Then this alternative statistic is more robust to outliers, more robust to the small component of the distribution that has extreme values, and increases the variance if they are not 'taken care of'.

Example with the above distribution. Let's consider a sample of size 10 and we compute the MLE as the mean of the sample and the alternative by considering only the mean of the middle 6 values. Let's see how do these differ in distribution/error:

### function to compute estimate in two different ways
get_sample = function() {
  ### geneare data
  n = 10
  sigma = 10^rbinom(n,1,0.25) ### mixture distribution 0.25 
          ### part sigma = 10 and 0.75 part sigma = 1 
  x = rnorm(n,0,sigma)
  
  ### compute estimates
  est1 = mean(x)
  est2 = mean(x[order(x)][3:8]) ### use only values 3 to 8 
                                ### (deleting outer 20%)
  return(c(est1,est2))
}

### compute the estimates
set.seed(1)
x <- replicate(10^4,get_sample()) 

### plot the histograms
layout(matrix(1:2,2))
hist(x[1,], breaks = seq(-10,10,0.1), xlim = c(-6,6), 
            freq = 0, xlab = "estimator value", main =  
            "distribution of estimated based on sample mean")
hist(x[2,], breaks = seq(-10,10,0.1), xlim = c(-6,6), 
        freq = 0, xlab = "estimator value", 
        main = "distribution of estimator based on mean of 6 
                 middle values ")

The MLE is often an estimator that has the lowest variance or performs sufficiently if the ideal conditions are true. But when the assumed distribution (on which this statement of low variance is based) is only slightly perturbed (but with values of large magnitude) then this can already result in a large variance for the MLE.

Note 1: It also depends on what sort of MLE you have. For instance, when we estimate the mean of a distribution and the distribution is a Gaussian distribution then the MLE is the mean of the sample, and as you see in the example above, the mean is not very robust against small perturbations. But when the distribution is a Laplace distribution then the MLE is the median of the sample, and this will be more robust against small perturbations.

Note 2: In the example above we simply excluded the bottom and top 20% from the sample. But robust estimators are not that simple. It is a complex and large field. For instance, what if we have only had positive outliers, then our discarding of the bottom part makes the estimate biased? And how much should we discard? There are many considerations that go into constructing a robust estimator (and sometimes it is a bit art instead of science, but the example shows the idea of why it generally works).

Answer 2

Let's consider a simple example for computing MLE of mean from a small sample dataset: $X=\{1, 2, 5, 10, 4, 8\}$.

Let's assume the data is generated with a Poisson distribution with rate $\lambda$ and we aim at finding the MLE of $\lambda$.

Now, the likelihood $L(\lambda|x_1,x_2,\ldots,x_n)=\prod\limits_{i=1}^{n}P(X=x_i)=\prod\limits_{i=1}^{n}\dfrac{e^{-\lambda}\lambda^{x_i}}{x_i!}$

s.t., the log likelihood $l(\lambda) = \sum\limits_{i=1}^{n}(-\lambda)+x_iln(\lambda)-ln(x_i!)=-n\lambda+ln(\lambda)\sum\limits_{i=1}^{n}x_i-\sum\limits_{i=1}^{n}ln(x_i!)$

$\implies \dfrac{\partial l}{\partial\lambda}=-n+\dfrac{\sum\limits_{i=1}^{n}x_i}{\lambda}=0$ at maximum

$\implies \hat{\lambda}_{MLE}=\dfrac{\sum\limits_{i=1}^{n}x_i}{n}$

Here, $n=6$, $\hat{\lambda}_{MLE}=5$, i.e., the sample mean.

Now, let's consider an outlier point $1000$ gets added to the dataset. Now, the MLE changes to $147.1429$, which clearly overfits to the noise in the data.