How to name and treat such a distribution of extreme values / outliers?

Question

In this sample data I have 4600 teenagers with their telephone calling frequency per week. Most of them are "normal" (max 5) but some are extreme with up to 75 calls per week.

I learned to visualize outliers via a box plot. But this does not work here.

The question is how to treat values like this?

1. How do I visualize them?

2. Are there other ways, compared with a box-plot (Whisker-Line?), to define outliers (and/or extreme values) in a distribution like this?

In this example of course "extreme" or "abnormal" values could be defined based on psychological research about phone addicted people. ;) But my question is about statistics and not about psychology. I want to know what is "extreme" compared with the whole sample.

Here is also a histogram for that data.

And this is the Python (with pandas) code to generate that data.

#!/usr/bin/env python3
import pandas as pd
import random as rd

rd.seed(7)
# "normal" call frequence (80 persons)
freq = rd.choices(range(0, 5), k=4400)
# "high" call frequences (80 persons)
freq.extend(rd.choices(range(26, 55), k=100))
freq.extend(rd.choices(range(56, 75), k=100))

df = pd.DataFrame(
    {
        'PERSON': range(4600),
        'PHONE_CALLS': freq,
    }
)

#df.PHONE_CALLS.plot.box().figure.show()
#df.PHONE_CALLS.plot.hist().figure.show()

Answer 1

This does not seem like a case where outliers are in a sense mistakes/errors, but rather a pattern where a subset of observations just happens to have very different values than the majority of the population. I.e. this is the kind of "outlier" that tells you that there is probably more going on than a simple single Poisson distribution and that some records are somehow different than the others. If you have important traits that explain this (e.g. perhaps all is explained by a "parents allow phone usage for reasons other than emergencies" feature or something else), then making a model that uses these features might be the obvious approach.

However, if you do not directly have features that explain the distribution, then one way to approach this is to consider this as a mixture distribution. I.e. there are really two (or more) distributions that get mixed together to get your overall population distribution. In the background, there's a latent (unobserved) trait such as "heavy phone user (y/n)". You could specify some plausible distribution (e.g. Poisson or negative binomial) and assume that there's, say, two mixture components that have distinct parameters. Such a model set-up then lets you estimate the probability that each user is in each component distribution.

Or, you could have not a latent binary feature, but rather a continuous one (=a random effects model, such as a random effects Poisson model).

Answer 2

There is naturally a large literature on outliers and many threads here. It is not especially cynical, I hope, to give a one sentence summary that if outliers are not obvious (as I consider they are in the question) then their identification is necessarily arbitrary. The use of box plots to make decisions about outliers goes far beyond what John W. Tukey intended in the 1970s when proposing his particular conventions for what should be shown.

More generally, what is an outlier depends crucially on what is expected. A common pattern in my experience is that apparent outliers are not outliers at all -- if considered on a scale and with respect to a distribution that makes sense for the problem.

The question of visualization is interesting and important. In principle, the fraction of outliers -- which appear sufficiently numerous to be regarded as a distinct subset, so I agree strongly with regarding this as a mixture problem -- could be anywhere from almost 25% to a much much smaller fraction.

A variant on a box plot than can work well is a quantile-box plot, whether the quantile plot and the box plot are side by side, as in the examples I am going to show, or superimposed. Although the term quantile-box plot has been used in other ways, I go back here, at least in spirit, to a proposal by Emanuel Parzen in 1979. Some references are given in What's the history of box plots, and how did the "box and whiskers" design evolve?

I am not clear why we are being shown a simulation rather than the data.

Without attempting a close fake of the graph in the question, I knocked up a mixture that is a discrete uniform on integers 1 to 4 and random draws from a normal distribution with mean 50 and SD 10. The graphs below show situations (left) where the high callers are 20% of the distribution and (right) where the high callers are about 1% of the distribution.

Regardless of precisely what graph is used, the reader should be told how many callers are in each large group. Also, a transformed scale (square root of number of calls or logarithm of number of calls (plus 1 if zeros are present)) might help and not hinder visualization. Indeed, such a transformed scale would make it easier to see "inside the box" and note the detail of how many occurrences there were of 0, 1, 2, 3, 4, ... in the data.

Detail: Once it's been decided to show all the data the oversold rule "whiskers extend to the outermost points within 1.5 IQR of the nearer quartile" can be ignored. Here the whiskers extend to the extremes and are redundant given the quantile plot, but do no harm.

Detail: I don't use Python (hope to learn some one wet week with nothing else to do) so don't suggest code. Something like this graph shouldn't be difficult in any decent software.