1
$\begingroup$

I am reading this lecture notes on using the MLEs from other distributions (as Laplace) rather than a Gaussian when dealing with outliers. The lecture notes came from Oxford University: https://www.cs.ox.ac.uk/people/varun.kanade/teaching/ML-MT2016/lectures/lecture03.pdf.

Probabilistically, we may view this as follows: the Gaussian distribution has very light ‘tails’, i.e., there is very little probability mass a couple of standard deviations away from the mean. Thus under this model outliers are very very unlikely and so the model will not treat them as such and try to fit a model that accounts for them rather than ignoring them. Instead, we can model the noise using a distribution that has heavier tails.

My question is this: Seeing that a Laplace distribution has a heavier tail compared to Gaussian, isn't it true though that there will be even less 'mass' on the tails of Laplace a few standard deviations from its mean? (I am just going by the logic that is presented by the paragraph above, which came from the lecture notes.)

I understand the motivation why there is a need to look for alternatives than the Gaussian when dealing with outliers, but I fail to understand how Laplace is able to solve this problem.

$\endgroup$
5
  • $\begingroup$ Related? $\endgroup$
    – Dave
    Commented Dec 23, 2022 at 7:23
  • 1
    $\begingroup$ "Seeing that a Laplace distribution has a heavier tail compared to Gaussian, isn't it true though that there will be even less 'mass' on the tails of Laplace a few standard deviations from its mean?" -- Can you clarify what logic you used to arrive at this? I followed the information from the linked notes but I cannot discern how you arrived at this conclusion. $\endgroup$
    – Glen_b
    Commented Dec 23, 2022 at 10:01
  • $\begingroup$ @Glen_b In the case of the Gaussian distribution (lets say Normal(0,1)), 99.7% of the data has already been observed. This means the probability of observing beyond the positive third standard deviation would be 0.15. Now in the case of the Laplace distribution, since it has a heavier tail, then the probability of observing beyond the positive third standard deviation would be less than 0.15, isn't it? I still don't understand how Laplace is able to do the job. $\endgroup$
    – cgo
    Commented Dec 23, 2022 at 10:49
  • 1
    $\begingroup$ The proportion beyond 3 s.d.s above the mean is much higher for the Laplace (more than 5 times as much). $\endgroup$
    – Glen_b
    Commented Dec 23, 2022 at 11:29
  • $\begingroup$ That’s the gist of the link I posted, that outliers pump up the variance. $\endgroup$
    – Dave
    Commented Dec 23, 2022 at 13:42

2 Answers 2

2
$\begingroup$

Laplace is able to solve the problem because it is robust to outliers.

enter image description here

Check the figures above: observe how Laplace fares against outliers compared to that of Gaussian.

As explained in $\rm [I], $ heavy tails

assign higher likelihood to outliers, without having to perturb the straight line to “explain” them.

More importantly, as noted in the lecture note linked by OP, the mle involves the minimization of absolute difference while that of Gaussian involved quadratic penalization of the outliers.


Source of Figure & Reference:

$\rm [I]$ Machine Learning: A Probabilistic Perspective, Kevin P. Murphy, MIT Press, $2012, $ fig. $2.8, $ p. $40;$ sec. $7.4, $ pp. $223-224.$

$\endgroup$
2
$\begingroup$

To address the issue with the tails, note that the log-density of the Laplace is linear either side of the mean, while that of the Gaussian is quadratic.

Clearly, then, as you head out into the tails, eventually the density of the Laplace must be higher than for the Gaussian. To set the mean and variance to be the same, it is relatively simple to take the standard normal, and we then set $b=\sqrt{\frac12}$ for the Laplace. Equating the log-densities we get a quadratic to solve; the Laplace density is higher from about 2.339 standard deviations either side of the mean.

Plot of log-density of Laplace and Gaussian with zero mean and unit variance

This is what the document you're reading means by the Gaussian having 'less mass' on the tails.

You were comparing the entire probability beyond a specific point. In many ways that's a better way to compare tail weight of the distribution*.

For that way of comparing the two, the cdf (on the left) and survivor function (on the right) of the Laplace will be higher than those for the normal beyond about 1.68 standard deviations above the mean.


* though something else again makes a better way of comparing the way that data is weighted in the sense of a weighted mean, say -- not that it's critical for the present point

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.