10
$\begingroup$

I'm new to statistics and currently learning abot MLE.

Some of the papers I read: Robust Graph Embedding with Noisy Link Weights mentioned MLEs are suspectible to contamination in data, but didn't make any further mention about it.

So I would like to know the reason for this with clear and detailed explains.

$\endgroup$
3
  • 1
    $\begingroup$ Because it is not robust. $\endgroup$ Oct 5, 2021 at 6:18
  • 6
    $\begingroup$ As a general claim it would be hard to support -- it depends on the MLE and what kind of robustness is intended; some MLEs correspond to fairly common choices of robust estimators of location (low breakdown/ low gross-error sensitivity). Unless the scope of the claim is limited in some way I would say it's not necessarily the case. $\endgroup$
    – Glen_b
    Oct 5, 2021 at 23:47
  • 3
    $\begingroup$ As a remark, many papers that justify not to use MLE only consider the MLE under the assumption of the normal distribution. In this case, it can be true that the estimation of the parameters of the distribution can be strongly inluenced by outliers, but MLE is not restricted to Gaussian pdf. $\endgroup$
    – radouxju
    Oct 6, 2021 at 12:43

2 Answers 2

18
$\begingroup$

Measurements do not always show ideal behavior and the presumed underlying distribution that is used for the maximum likelihood estimate (MLE) is often not the distribution of the measurements.

measurements not ideal

For instance, in the graph above the distribution of the measurements is a mixture of two Gaussian distributions 25% with $\sigma = 10$ and 75% with $\sigma = 1$.

(so for whatever reason the distribution is not ideal, either it is because the population is not ideal or it is because the measurements are not perfect)

This component with large variation will increase the sampling variance (and inaccuracy/error) of the estimator a lot.

Instead of using the MLE (which in a simple case, like estimating the mean of the population, boils down to the average/mean value of the sample), one could use a statistic that filters a few of the extreme values from the sample. This would greatly reduce the variation of the statistic. Then this alternative statistic is more robust to outliers, more robust to the small component of the distribution that has extreme values, and increases the variance if they are not 'taken care of'.

Example with the above distribution. Let's consider a sample of size 10 and we compute the MLE as the mean of the sample and the alternative by considering only the mean of the middle 6 values. Let's see how do these differ in distribution/error:

demonstration of difference

### function to compute estimate in two different ways
get_sample = function() {
  ### geneare data
  n = 10
  sigma = 10^rbinom(n,1,0.25) ### mixture distribution 0.25 
          ### part sigma = 10 and 0.75 part sigma = 1 
  x = rnorm(n,0,sigma)
  
  ### compute estimates
  est1 = mean(x)
  est2 = mean(x[order(x)][3:8]) ### use only values 3 to 8 
                                ### (deleting outer 20%)
  return(c(est1,est2))
}

### compute the estimates
set.seed(1)
x <- replicate(10^4,get_sample()) 

### plot the histograms
layout(matrix(1:2,2))
hist(x[1,], breaks = seq(-10,10,0.1), xlim = c(-6,6), 
            freq = 0, xlab = "estimator value", main =  
            "distribution of estimated based on sample mean")
hist(x[2,], breaks = seq(-10,10,0.1), xlim = c(-6,6), 
        freq = 0, xlab = "estimator value", 
        main = "distribution of estimator based on mean of 6 
                 middle values ")

The MLE is often an estimator that has the lowest variance or performs sufficiently if the ideal conditions are true. But when the assumed distribution (on which this statement of low variance is based) is only slightly perturbed (but with values of large magnitude) then this can already result in a large variance for the MLE.

Note 1: It also depends on what sort of MLE you have. For instance, when we estimate the mean of a distribution and the distribution is a Gaussian distribution then the MLE is the mean of the sample, and as you see in the example above, the mean is not very robust against small perturbations. But when the distribution is a Laplace distribution then the MLE is the median of the sample, and this will be more robust against small perturbations.

Note 2: In the example above we simply excluded the bottom and top 20% from the sample. But robust estimators are not that simple. It is a complex and large field. For instance, what if we have only had positive outliers, then our discarding of the bottom part makes the estimate biased? And how much should we discard? There are many considerations that go into constructing a robust estimator (and sometimes it is a bit art instead of science, but the example shows the idea of why it generally works).

$\endgroup$
8
$\begingroup$

Let's consider a simple example for computing MLE of mean from a small sample dataset: $X=\{1, 2, 5, 10, 4, 8\}$.

Let's assume the data is generated with a Poisson distribution with rate $\lambda$ and we aim at finding the MLE of $\lambda$.

Now, the likelihood $L(\lambda|x_1,x_2,\ldots,x_n)=\prod\limits_{i=1}^{n}P(X=x_i)=\prod\limits_{i=1}^{n}\dfrac{e^{-\lambda}\lambda^{x_i}}{x_i!}$

s.t., the log likelihood $l(\lambda) = \sum\limits_{i=1}^{n}(-\lambda)+x_iln(\lambda)-ln(x_i!)=-n\lambda+ln(\lambda)\sum\limits_{i=1}^{n}x_i-\sum\limits_{i=1}^{n}ln(x_i!)$

$\implies \dfrac{\partial l}{\partial\lambda}=-n+\dfrac{\sum\limits_{i=1}^{n}x_i}{\lambda}=0$ at maximum

$\implies \hat{\lambda}_{MLE}=\dfrac{\sum\limits_{i=1}^{n}x_i}{n}$

Here, $n=6$, $\hat{\lambda}_{MLE}=5$, i.e., the sample mean.

Now, let's consider an outlier point $1000$ gets added to the dataset. Now, the MLE changes to $147.1429$, which clearly overfits to the noise in the data.

$\endgroup$
2
  • 1
    $\begingroup$ True, robust distributions allowing small probabilities for outliers reduce overfitting, but I think OP wanted to see how MLE is susceptible to noise for some distribution, since it's entirely dependent on data and we don't have any regularizer like prior to control overfitting to some extent. It's just a small example to demonstrate a particular case and not a generic proof anyway. $\endgroup$ Oct 5, 2021 at 6:52
  • $\begingroup$ @Tim, the OP mentions data contamination, so the explanation should address that kind of setting. Replacing the likelihood function of uncontaminated data with one that fits contaminated data would miss the point. I think this answer does a decent job, though it could be more explicit in its verbal explanation in the last paragraph. $\endgroup$ Oct 5, 2021 at 7:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.