Which are outliers?

Question

I am in the process of solving a Machine Learning challenge, and I want to do it the right way.

I did some exploratory data analysisand I wanted to check the distribution of the data. As displayed in the figure below, the box plots of each of the numerical continuous dataset features.
According to my understanding, several features do contain outliers, but after reading some articles, I encountered cases where outliers are mistaken for a Pareto distribution, and removing them leads to information loss.

Could you please explain which contain and which do not contain outliers?

Answer 1

Outliers, like beauty and much else, are often in the eye of the beholder.

Identifying outliers is often treated as if it were a matter of knowing that your data are a mix of gazelles (regular beasts) and giraffes (probably large, awkward and unwanted), so the problem is to tell which is which. In real applications, it may not be obvious that there are two such distinct subsets, even in abstract principle. And even if the principle is accepted, it can be hard in practice to tell them apart: imagine how you would proceed if the variables were height, weight, and number of legs and there are known to be young giraffes in the data.

Box plots are routinely misused in this context. First of all, although box plots were (re-)introduced by John W. Tukey in the 1970s, he himself played with several different conventions, and later people haven't stopped inventing different variations -- which could be worthwhile experiment. Unfortunately users often fail to explain in reports which detailed rules their software uses -- which is not helpful.

Here I guess that the convention being followed is one Tukey introduced: plot data points individually whenever any is more than 1.5 IQR from the nearer quartile. Here IQR is interquartile range or upper quartile MINUS lower quartile. Tukey himself regarded this just as a way of identifying data points that need to be thought about, and not at all as a reliable criterion for identifying points that should be rejected. Indeed, sometimes outliers (and skewness) are a signal that data should be analysed on a transformed scale and/or that they need resistant or robust summaries. .

I won't try to say which of your data points are outliers, because

1. The whole idea that some points are expected to be BAD outliers or problematic, and the rest GOOD, needs to be argued in each case. For example, in banking some transactions are rogue and most are honest, so there is strong concern to identify rogues, which is rarely easy. In many other fields, strongly skewed distributions are entirely expected and the larger outliers are known extremes (the Amazon, or Amazon, say).

2. There are no outliers except in relation to an implicit or explicit model of what is expected. If one distribution is expected to be (roughly) normal and another (roughly) Poisson or anything else, I have quite different expectations of whether outliers are likely and how to detect them. (This is the kind of point made in the question with the example of a Pareto distribution.) Here the data are just labelled X without even a signal for whether they are versions of the same (kind of) variable or whether they are quite different.

3. Box plots alone are inadequate to make the decision even if #1 and #2 are satisfied. I'd much prefer dot or quantile plots. I'd also want to know the number of data points, which is impossible to discern precisely. But by failing to show simple patterns box plots can signal that you need another and a deeper look. Some of the box plots above are degenerate in that median and quartiles are identical, or very close, so that the box collapses to a line: in that case every other data point is plotted individually. (If the IQR is zero, every point beyond the quartiles must be plotted individually!) Such a pattern could be anything from the quirks of small samples to some special situation: the box plots alone cannot do more than flag an oddity. Specifically X27 X31 X33 spring out as demanding careful scrutiny.

4. Specifically the numerical labels for these data suggest that the data were standardized first (likely by (value MINUS mean) / SD): this pre-processing is over-valued and inhibits or prohibits reading off basic characteristics of the data, such as

• the data being all positive

• the data being all positive or zero

• the data being positive, zero or negative

• whether only certain values occur (e.g. discrete grades or counts that are integers). This is one of the worst problems, as box plots are often hard work with such data. Repeated values plot as one symbol or contribute to compressed boxes. X33 seems possibly an example as there are perhaps only 3 distinct values; they may even be nominal codes with no intrinsic meaning.

Answer 2

You have two questions

What are outliers?

Which are outliers?

Should outliers be removed?

According to my understanding, several features do contain outliers, but after reading some articles, I encountered cases where outliers are mistaken for a Pareto distribution, and removing them leads to information loss.

There can be different interpretations of 'outliers'

• Outliers as extreme cases within a population: Outliers can be referred to the cases within a population in the outer regions. For instance the book 'Outliers' from Malcolm Gladwell speaks about outliers as extreme cases, cases that are relatively far from the mean.

• Outliers as measurement errors: In statistics and experimental science/research, outliers can also refer to erroneous measurements. Data points that are not representing the population of interest because of some error in the sampling method or experimental execution.

The latter type of outliers, experimental errors, you might want to remove.

But the former type of outlier, extreme cases among the population of measurements, are not always the same as these latter types.

Wrong measurements/sampling

All measurements have some error* due to small variations in the population and experiments. But some measurements introduce error to a much larger extent and are a bad representation of the population that is being sampled or used for experimentation.

It is these types of measurement or sample errors that we may want to remove from the data because they cause bias and/or high variance in estimates of population parameters.

^{*It might be better to speak about 'disturbance' rather than 'error'. The variation of measurements is an error in a context of a physics or chemistry experiment, where some fixed value is being measured. But, in psychology, nutrition, health, econometrics, the variations are natural since there is no fixed value and a varied population is being investigated. We do not want to filter out all the errors if these errors are what is representing the population. Discarding data in preprocessing is about the wrong measurements.}

For example, imagine some experiment that determines and compares the growth rate of two different strains of bacteria.

• Some of the experiments might have a bias.

  e.g. the temperature was not set correctly causing a consistent/systematic error.

• Some of the experiments might have a large variance

  e.g. some batches of bacteria were not kept under optimal conditions and have a relatively large variance in the measurements.

Outliers as a way to detect erroneous measurements

Detecting these cases of error in measurements and sampling is part of preprocessing the data and data cleansing.

One way to detect the erroneous measurements is by finding out the outliers in the first sense, finding extreme cases in the population of measurements.

• Outliers are extreme cases in a population.
• When outliers in a population of measurements can be considered as wrongful measurements/sampling, i.e. they are a part of the measurements/sampling that is not representing the population of interest, then they could be removed.

Should they be removed?

The choice to remove the outliers depends on whether they are likely wrong measurements/sampling and whether they are likely to influence the statistical analysis.

• Outliers in the population of measurements may not necessarily be equal to wrong measurements. If you are removing outliers too aggressively, then you might also remove data points that are not wrong measurements/sampling. The result is that the amount of data is decreasing. And also you can potentially introduce bias by having a selective/subjective sampling process that wrongly represents the population that is under investigation because there is too much censoring of extreme data points. Removing outliers in the measurements should be a balance between advantages and disadvantages.
• Besides removing outliers, you can also perform an analysis that is more robust towards outliers. For instance, you can consider the wrong measurements as 'correct' measurements (since all measurements have some 'error') but just with a different type of error. One alternative way to deal with it is, for instance, using a different cost function in regression (e.g. use absolute deviations instead of squared deviations).

In your example, it is difficult to say because there is no information about the data (what does it represent?). But we see so many 'outliers' that we may wonder whether they are not just naturally part of the population. They are outliers in the first sense (extreme data points) but that does not necessarily make them wrong measurements.

Answer 3

An example to illustrate the points made well above.

On most days, the Dow Jones Index increases slightly. About once a decade it drops by more than 20%. An investor who ignores the "outliers" will not do well.