4
$\begingroup$

I am in the process of solving a Machine Learning challenge, and I want to do it the right way.

I did some exploratory data analysisand I wanted to check the distribution of the data. As displayed in the figure below, the box plots of each of the numerical continuous dataset features.
According to my understanding, several features do contain outliers, but after reading some articles, I encountered cases where outliers are mistaken for a Pareto distribution, and removing them leads to information loss.

box plots of numerical features

Could you please explain which contain and which do not contain outliers?

$\endgroup$
9
  • $\begingroup$ Kaggle Corporate Bankruptcy Prediction 2021; Bankruptcy data from the Taiwan Economic Journal for the years 1999–2009, yes? $\endgroup$
    – smci
    Jan 27 at 15:30
  • 2
    $\begingroup$ Ah ok. It helps if you tell us what type of data you think this is (stocks? weather? biology?) etc., to develop some intution about correlations or dependencies. The boxplots for X27, X31, X33 don't show much, better to histogram the few discrete values (/intervals). $\endgroup$
    – smci
    Jan 27 at 15:51
  • 9
    $\begingroup$ Not my problem, but expecting a good data analysis without a context is a bizarre exercise. This perhaps exposes a cultural difference between (some kinds of) machine learning and (what I regard as good practice in) statistics. $\endgroup$
    – Nick Cox
    Jan 27 at 16:00
  • 1
    $\begingroup$ @RockPaperLz-MaskitorCasket Leo Breiman's paper on two cultures is easy to find and 21 years young. Others may be able to give references more recent and as good or better. $\endgroup$
    – Nick Cox
    Jan 28 at 9:10
  • 1
    $\begingroup$ The starkest contrasts are between one approach done badly and the other done well (and so essentially unfair). As someone more on the statistical side than in machine learning, I am sometimes gobsmacked by some mentions on CV of machine learning, especially if someone has picked up a sense that in essence data PLUS code $\implies$ results, without any need to know context, think about goals or explore data. But mainstream statistics thoughtlessly applied can lead to nonsense too, as in applying techniques utterly unsuited to the data or ridiculous over-emphasis on significance tests. $\endgroup$
    – Nick Cox
    Jan 28 at 11:53

3 Answers 3

38
$\begingroup$

Outliers, like beauty and much else, are often in the eye of the beholder.

Identifying outliers is often treated as if it were a matter of knowing that your data are a mix of gazelles (regular beasts) and giraffes (probably large, awkward and unwanted), so the problem is to tell which is which. In real applications, it may not be obvious that there are two such distinct subsets, even in abstract principle. And even if the principle is accepted, it can be hard in practice to tell them apart: imagine how you would proceed if the variables were height, weight, and number of legs and there are known to be young giraffes in the data.

Box plots are routinely misused in this context. First of all, although box plots were (re-)introduced by John W. Tukey in the 1970s, he himself played with several different conventions, and later people haven't stopped inventing different variations -- which could be worthwhile experiment. Unfortunately users often fail to explain in reports which detailed rules their software uses -- which is not helpful.

Here I guess that the convention being followed is one Tukey introduced: plot data points individually whenever any is more than 1.5 IQR from the nearer quartile. Here IQR is interquartile range or upper quartile MINUS lower quartile. Tukey himself regarded this just as a way of identifying data points that need to be thought about, and not at all as a reliable criterion for identifying points that should be rejected. Indeed, sometimes outliers (and skewness) are a signal that data should be analysed on a transformed scale and/or that they need resistant or robust summaries. .

I won't try to say which of your data points are outliers, because

  1. The whole idea that some points are expected to be BAD outliers or problematic, and the rest GOOD, needs to be argued in each case. For example, in banking some transactions are rogue and most are honest, so there is strong concern to identify rogues, which is rarely easy. In many other fields, strongly skewed distributions are entirely expected and the larger outliers are known extremes (the Amazon, or Amazon, say).

  2. There are no outliers except in relation to an implicit or explicit model of what is expected. If one distribution is expected to be (roughly) normal and another (roughly) Poisson or anything else, I have quite different expectations of whether outliers are likely and how to detect them. (This is the kind of point made in the question with the example of a Pareto distribution.) Here the data are just labelled X without even a signal for whether they are versions of the same (kind of) variable or whether they are quite different.

  3. Box plots alone are inadequate to make the decision even if #1 and #2 are satisfied. I'd much prefer dot or quantile plots. I'd also want to know the number of data points, which is impossible to discern precisely. But by failing to show simple patterns box plots can signal that you need another and a deeper look. Some of the box plots above are degenerate in that median and quartiles are identical, or very close, so that the box collapses to a line: in that case every other data point is plotted individually. (If the IQR is zero, every point beyond the quartiles must be plotted individually!) Such a pattern could be anything from the quirks of small samples to some special situation: the box plots alone cannot do more than flag an oddity. Specifically X27 X31 X33 spring out as demanding careful scrutiny.

  4. Specifically the numerical labels for these data suggest that the data were standardized first (likely by (value MINUS mean) / SD): this pre-processing is over-valued and inhibits or prohibits reading off basic characteristics of the data, such as

  • the data being all positive

  • the data being all positive or zero

  • the data being positive, zero or negative

  • whether only certain values occur (e.g. discrete grades or counts that are integers). This is one of the worst problems, as box plots are often hard work with such data. Repeated values plot as one symbol or contribute to compressed boxes. X33 seems possibly an example as there are perhaps only 3 distinct values; they may even be nominal codes with no intrinsic meaning.

$\endgroup$
6
  • 5
    $\begingroup$ +1. Marvelous discussion. I have concerns about (2), though, because I think it is the opposite of what Tukey maintained. Many of his methods automatically identify "outliers" (and even "far outliers"). The difference is that outlier identification becomes a tool to help analyze the data, perhaps by suggesting ways to re-express or smooth them. Key elements of the underlying conception are (1) that "outliers" often occur, (2) they do not necessarily reflect "rogue" or "bad" data, and (3) are treated as adding useful detail to the gross descriptions afforded by summary statistics. $\endgroup$
    – whuber
    Jan 26 at 16:17
  • 5
    $\begingroup$ @whuber I don't disagree and I've modified or added wording to give a more nuanced summary. I am particularly disturbed at a meme I find on CV: points more than 1.5 IQR from quartiles should just be deleted! $\endgroup$
    – Nick Cox
    Jan 26 at 18:53
  • $\begingroup$ Thank you Nick for the outstanding explanation. However, I'd like to ask one more question, the data contains 400 samples, 100 of which is not labeled, so the task is to predict the labels of the remaining 100. The data consists, of both num. continuous values and cat. nominal values. The latter will be encoded. My question is the following, concerning the nominal data, if you encounter an attribute which consists of two or three values (e.g. V1, V2, V3) but among all instances, it only contains one instance of V3, would you replace this with the mode? since it could be a data entry mistake. $\endgroup$ Jan 27 at 11:10
  • 1
    $\begingroup$ Or it could be a genuine but unusual value. I find it hard to tell you what's right there. For example, I would look at the values of other variables for the same observation to see whether there was a strong impression of something very unusual going on. Again, the context that's missing here is important. If the data are people filling in questionnaires that's one thing. If the data are produced by an automatic weather station, that's another. $\endgroup$
    – Nick Cox
    Jan 27 at 11:29
  • $\begingroup$ As a matter of curiosity only, which software was used for the box plots? $\endgroup$
    – Nick Cox
    Jan 27 at 11:30
8
$\begingroup$

You have two questions

What are outliers?

Which are outliers?

Should outliers be removed?

According to my understanding, several features do contain outliers, but after reading some articles, I encountered cases where outliers are mistaken for a Pareto distribution, and removing them leads to information loss.

There can be different interpretations of 'outliers'

  • Outliers as extreme cases within a population: Outliers can be referred to the cases within a population in the outer regions. For instance the book 'Outliers' from Malcolm Gladwell speaks about outliers as extreme cases, cases that are relatively far from the mean.

  • Outliers as measurement errors: In statistics and experimental science/research, outliers can also refer to erroneous measurements. Data points that are not representing the population of interest because of some error in the sampling method or experimental execution.

The latter type of outliers, experimental errors, you might want to remove.

But the former type of outlier, extreme cases among the population of measurements, are not always the same as these latter types.

Wrong measurements/sampling

All measurements have some error* due to small variations in the population and experiments. But some measurements introduce error to a much larger extent and are a bad representation of the population that is being sampled or used for experimentation.

It is these types of measurement or sample errors that we may want to remove from the data because they cause bias and/or high variance in estimates of population parameters.

*It might be better to speak about 'disturbance' rather than 'error'. The variation of measurements is an error in a context of a physics or chemistry experiment, where some fixed value is being measured. But, in psychology, nutrition, health, econometrics, the variations are natural since there is no fixed value and a varied population is being investigated. We do not want to filter out all the errors if these errors are what is representing the population. Discarding data in preprocessing is about the wrong measurements.

outliers

For example, imagine some experiment that determines and compares the growth rate of two different strains of bacteria.

  • Some of the experiments might have a bias.

    e.g. the temperature was not set correctly causing a consistent/systematic error.

  • Some of the experiments might have a large variance

    e.g. some batches of bacteria were not kept under optimal conditions and have a relatively large variance in the measurements.

Outliers as a way to detect erroneous measurements

Detecting these cases of error in measurements and sampling is part of preprocessing the data and data cleansing.

One way to detect the erroneous measurements is by finding out the outliers in the first sense, finding extreme cases in the population of measurements.

  • Outliers are extreme cases in a population.
  • When outliers in a population of measurements can be considered as wrongful measurements/sampling, i.e. they are a part of the measurements/sampling that is not representing the population of interest, then they could be removed.

Should they be removed?

The choice to remove the outliers depends on whether they are likely wrong measurements/sampling and whether they are likely to influence the statistical analysis.

  • Outliers in the population of measurements may not necessarily be equal to wrong measurements. If you are removing outliers too aggressively, then you might also remove data points that are not wrong measurements/sampling. The result is that the amount of data is decreasing. And also you can potentially introduce bias by having a selective/subjective sampling process that wrongly represents the population that is under investigation because there is too much censoring of extreme data points. Removing outliers in the measurements should be a balance between advantages and disadvantages.
  • Besides removing outliers, you can also perform an analysis that is more robust towards outliers. For instance, you can consider the wrong measurements as 'correct' measurements (since all measurements have some 'error') but just with a different type of error. One alternative way to deal with it is, for instance, using a different cost function in regression (e.g. use absolute deviations instead of squared deviations).

In your example, it is difficult to say because there is no information about the data (what does it represent?). But we see so many 'outliers' that we may wonder whether they are not just naturally part of the population. They are outliers in the first sense (extreme data points) but that does not necessarily make them wrong measurements.

$\endgroup$
1
  • 1
    $\begingroup$ +1 Thank you for taking the time to answer my question. You made it clear that the context of data plays an essential role to decide whether to handle extreme data points as anomalous or not. $\endgroup$ Jan 27 at 11:20
4
$\begingroup$

An example to illustrate the points made well above.

On most days, the Dow Jones Index increases slightly. About once a decade it drops by more than 20%. An investor who ignores the "outliers" will not do well.

$\endgroup$
2
  • 3
    $\begingroup$ Not an answer to the question, but rather a comment on Nick's answer? $\endgroup$
    – marianoju
    Jan 27 at 10:38
  • 1
    $\begingroup$ The point to be made here is, roughly, "Do outliers represent information of interest, or are they just the result of insufficient sampling?" $\endgroup$ Jan 27 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.