Can you remove outliers if they are less than 10% of the datapoints? [duplicate]

Question

I am currently attending my first data analysis class and we do some simple hypothesis tests like t test etc. Our teacher told us that we can remove outliers, as long as they are not more that the 10% of our sample size n. Is this accurate? It seems too methodological to me and I don't understand why we handle every situation like this. What is the general approach?

Answer 1

The problem here is how to define an outlier. Personally I would only remove an outlier if I thought there was a good reason to believe it was the result of, for example, a measurement error and wasn't representative of the underlying data generating process. I wouldn't remove them if they were simply an unusual event, as that would lead to underestimation of uncertainties or variances, which can also be important in some applications.

I would define an outlier as a data point that can't be "explained" with reasonable probability by a model that otherwise explained the dataset well. This means an outlier is relative to your model of the data generating process. If your model of the data generating process was wrong, then you might incorrectly remove an outlier that was not an overly unusual observation under a better model of the data. This could cause you to draw incorrect conclusions.

For example, some time ago I wrote a comment on a paper about estimating the body mass of dinosaurs based on allometry. In allometry, it is common to use a linear regression model on logarithmic axes of body measurements (in this case, long bone circumference and body mass) because the relationships are often described by a power law, with constant relative error. Packard et al. however argued for a power law model with constant absolute error (which is biologically implausible in this case). The diagram below shows the traditional (a) and (b) and absolute error (c) and (d) allometric models of the data:

Under the traditional allometric model (b) the elephant is close to being an outlier. Under the absolute error model (d) the hippopotamus (the next heaviest mammal) clearly is an outlier, being substantially outside the $2\sigma$ prediction interval. In this case, neither mammal is actually an outlier, the measurements are reliable, it is just that elephants are unusually active and have large leg bones for their weight, whereas hippos, being semi-aquatic, have relatively slender bones for their size.

So if you use simple univariate tests for outliers that are not based on an understanding of the data generating process, you may well end up rejecting datapoints that tell you important information.

In this case, Packard et al didn't treat the hippo or elephant as an outlier, but they did draw an incorrect conclusion (dinosaurs were only about half as heavy as previously thought) because their model was wrong (constant absolute error is biologically implausible). This meant that their model was dominated by a single observation, the heaviest mammal - the elephant. Had they deleted the elephant, they might have concluded that dinosaurs were substantially heavier than previously thought, because their model would be dominated by the next heaviest mammal - the hippopotamus.

We have to try and consider the data generating process before we take action, such as deleting outliers. Rules of thumb are useful sometimes, but you need to investigate whether they are appropriate for each particular project by using sensitivity analysis or diagnostic tests of the model.

Answer 2

As @NickCox wrote, there are many threads already discussing this.

Some remarks:

1. There is no agreed definition of what an outlier actually is. Some people talk about "outliers" as if they were always bad and problematic, but note that the fact that an observation appears outlying in the data does not mean that the observation is invalid and not informative. In fact in some situations outliers are most important (for example in insurance premium calculations). Also what is called an outlier will depend on an implicit model idea for the non-outlying observations (the answer of @DikranMarsupial illustrates this nicely).

2. Background information about the meaning of the data should always be taken into account. This includes potential reasons for observing outliers in the given situation, possible or plausible value ranges, and information about real life consequences from rightly or wrongly incorporating or removing outliers.

3. When considering to remove outliers, an attempt should be made to find out why an observation is outlying. In some situations one can find out from background knowledge (possible value ranges, information about the data collection process) that an observation is indeed erroneous, in which case it should be removed or corrected. In certain other situations one may find that outliers, although correct, are distinctively special in some respect, and it makes sense to analyse outliers and non-outliers separately (in case of uncertainty about what the outliers actually are and which and how many different processes are at work generating different groups of data, it may be useful to run a cluster analysis).

4. Regarding consequences, treatment of outliers should depend on knowledge of what is done afterwards with the data, and how sensitive this is to outliers. Note for example that an extreme outlier can move a mean outside the range of the non-outliers, which is inappropriate if the outlier is in fact bad, however in some situations this may be justified (for example if our observations are expenses and the outlier is just a very large expense). An extreme outlier hardly does any harm to the median, and generally there are so-called robust estimators, also for regression and other problems, that are robust against outliers, and therefore they don't require removing them. As alternative to the t-test, Wilcoxon rank tests are hardly affected by outliers. There are also tests comparing medians or tests based on robust Huber M-estimators. Although these don't test exactly the same null hypotheses (as mentioned elsewhere), they correspond to the same interpretation that the researcher may have in mind regarding the meaning of the results in many applications (e.g., in the two-sample case, that one sample gives systematically higher values than the other vs. about the same).

Answer 3

Your instinct is correct and your professor's advice is, at best, something that you can follow until you learn more. However, what I would give as a "first rule" is "never remove outliers unless they are data entry errors; if you have outliers, call someone who knows more stats than you do, at this point, but pay attention to outliers and try to figure them out."

You mention a t-test, and even a single outlier can make the results wrong. It's true, as Dikran said, that there's no general agreement on what an outlier is, but the more extreme the outlier, the more it will affect the t-test.

Later, you may learn about robust methods that account for outliers, but these procedures test different hypotheses (e.g. that the medians are different) and you may not want that.

This is a complex subject and at least one whole textbook is devoted solely to detecting and dealing with outliers.

Answer 4

Adding to the general chorus of answers, all mostly in the same direction (your instructor is giving you bad advice).

I personally dislike the term "outlier", because of its negative implication, implying we have to remove them. And I dislike even more your instructor's advice.

There are basically 2 types of observations (data points). The first kind is actual data, truly coming from your data generating process (DGP), maybe tainted by some noise, measurement uncertainty, etc. This is real data: it belongs to the DGP you are studying, and you have to keep it. Or there is erroneous data, coming from measurement process errors (including human errors). These are external to your DGP, are artificial, and ideally should be removed, if you can.

The first kind, you have to keep; after all, would you say that heavy tailed distributions have many outliers? No, they may generate extreme values, but that is in keeping with their underlying distributions. The DGP can not (by definition) produce "outliers"; it may produce noisy data, but that is the best you can do, per your experimental process.

The 2nd kind, you can (should?) try to remove (because they are polluting your data), but very carefully, and only if you have some demonstrable evidence. For example, suppose you are measuring shaft lengths, which you know should be more or less centered on 10 cm. If you see a measurement at 10.9, while a little too far for your liking (you would hope your machining process would be more precise than this), you can not exclude it as an "outlier". But if you see a 103 entry, you can be pretty sure that it probably truly was 10.3, and hence can exclude it (or correct it). But if you see a 13 cm entry, while alarming, you have no valid, demonstrable reason to exclude it (it could also have been 10.3 -missing 0-, but maybe not). And the fact that it represents less than 10% of all data points is certainly no valid reason (and note that if you distrust up to 10% of your data, you have much bigger problems to deal with your data collection setup).

So based on context, the specific characteristics of your measurements, their precision, the fundamental of the DGP, etc., you can (and probably should) remove/correct data points which are very likely to be "errors", based on credible evidence. Beyond that, you should leave the data alone. Any such attempt at removing so-called "outliers" is a subjective form of data tampering (let's remove all the data which does not conform to my predetermined conclusion, as long as it does not amount to more than 10% of the whole dataset). This is just torturing the data until it confesses to whatever you wanted it to say.

And when you encounter any such "extreme" observations, your only legitimate remedy is to collect more data. This additional data may reduce the impact of the so-called "outlier", or may in fact produce more "extreme" observations, proving that such observations are not so "outlying" after all...