Heavy vs light tail distributions when modelling with outliers

Question

I am reading this lecture notes on using the MLEs from other distributions (as Laplace) rather than a Gaussian when dealing with outliers. The lecture notes came from Oxford University: https://www.cs.ox.ac.uk/people/varun.kanade/teaching/ML-MT2016/lectures/lecture03.pdf.

Probabilistically, we may view this as follows: the Gaussian distribution has very light ‘tails’, i.e., there is very little probability mass a couple of standard deviations away from the mean. Thus under this model outliers are very very unlikely and so the model will not treat them as such and try to fit a model that accounts for them rather than ignoring them. Instead, we can model the noise using a distribution that has heavier tails.

My question is this: Seeing that a Laplace distribution has a heavier tail compared to Gaussian, isn't it true though that there will be even less 'mass' on the tails of Laplace a few standard deviations from its mean? (I am just going by the logic that is presented by the paragraph above, which came from the lecture notes.)

I understand the motivation why there is a need to look for alternatives than the Gaussian when dealing with outliers, but I fail to understand how Laplace is able to solve this problem.

Answer 1

To address the issue with the tails, note that the log-density of the Laplace is linear either side of the mean, while that of the Gaussian is quadratic.

Clearly, then, as you head out into the tails, eventually the density of the Laplace must be higher than for the Gaussian. To set the mean and variance to be the same, it is relatively simple to take the standard normal, and we then set $b=\sqrt{\frac12}$ for the Laplace. Equating the log-densities we get a quadratic to solve; the Laplace density is higher from about 2.339 standard deviations either side of the mean.

This is what the document you're reading means by the Gaussian having 'less mass' on the tails.

You were comparing the entire probability beyond a specific point. In many ways that's a better way to compare tail weight of the distribution*.

For that way of comparing the two, the cdf (on the left) and survivor function (on the right) of the Laplace will be higher than those for the normal beyond about 1.68 standard deviations above the mean.

---

* though something else again makes a better way of comparing the way that data is weighted in the sense of a weighted mean, say -- not that it's critical for the present point

Answer 2

Laplace is able to solve the problem because it is robust to outliers.

Check the figures above: observe how Laplace fares against outliers compared to that of Gaussian.

As explained in $\rm [I], $ heavy tails

assign higher likelihood to outliers, without having to perturb the straight line to “explain” them.

More importantly, as noted in the lecture note linked by OP, the mle involves the minimization of absolute difference while that of Gaussian involved quadratic penalization of the outliers.

---

Source of Figure & Reference:

$\rm [I]$ Machine Learning: A Probabilistic Perspective, Kevin P. Murphy, MIT Press, $2012, $ fig. $2.8, $ p. $40;$ sec. $7.4, $ pp. $223-224.$