Is it OK to remove outliers from data?

Question

I looked for a way to remove outliers from a dataset and I found this question.

In some of the comments and answers to this question, however, people mentioned that it is bad practice to remove outliers from the data.

In my dataset I have several outliers that very likely are just due to measurement errors. Even if some of them are not, I have no way of checking it case by case, because there are too many data points. Is it statistically valid than just to remove the outliers? Or, if not, what could be another solution?

If I just leave those points there, they influence e.g. the mean in a way that does not reflect reality (because most of them are errors anyway).

EDIT: I am working with skin conductance data. Most of the extreme values are due to artifacts like somebody pulling on the wires.

EDIT2: My main interest in analyzing the data is to determine if there is a difference between two groups

Answer 1

One option is to exclude outliers, but IMHO that is something you should only do if you can argue (with almost certainty) why such points are invalid (e.g. measurement equipment broke down, measurement method was unreliable for some reason, ...). E.g. in frequency domain measurements, DC is often discarded since many different terms contribute to DC, quite often unrelated to the phenomenon you are trying to observe.

The problem with removing outliers, is that to determine which points are outliers, you need to have a good model of what is or is not "good data". If you are unsure about the model (which factors should be included, what structure does the model have, what are the assumptions of the noise, ...), then you cannot be sure about your outliers. Those outliers might just be samples that are trying to tell you that your model is wrong. In other words: removing outliers will reinforce your (incorrect!) model, instead of allowing you to obtain new insights!

Another option, is to use robust statistics. E.g. the mean and standard deviation are sensitive to outliers, other metrics of "location" and "spread" are more robust. E.g. instead of the mean, use the median. Instead of standard deviation, use inter-quartile range. Instead of standard least-squares regression, you could use robust regression. All those robust methods de-emphasize the outliers in one way or another, but they typically do not remove the outlier data completely (i.e. a good thing).

Answer 2

I do not recommend excluding any outlier in the main analysis (unless you are really positive they are mistaken). You can do it in a sensitivity analysis, though, and compare the results of the two analyses. In science, often you discover new stuff precisely when focusing on such outliers.

To further elaborate, just think about the seminal Fleming's discovery of penicillin, based on the accidental contamination of his experiments with a mold:

http://www.abpischools.org.uk/page/modules/infectiousdiseases_timeline/timeline6.cfm?coSiteNavigation_allTopic=1

Looking at the near past or present, outlier detection is often used to guide innovation in biomedical sciences. See for instance the following articles (with some suitable R codes):

http://www.la-press.com/a-comparison-of-methods-for-data-driven-cancer-outlier-discovery-and-a-article-a2599-abstract?article_id=2599

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3394880/

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0102678

Finally, if you have reasonable grounds to exclude some data, you may do it, preferably in a sensitivity analysis, and not in the primary one. For instance you could exclude all values which are not biologically plausible (such as a temperature of 48 degrees Celsius in a septic patient). Similarly, you could exclude all first and last measurements for any given patient, to minimize movement artifacts. Take notice however that if you do this post-hoc (not based on a pre-specified criteria), this risks amounting to data massaging.

Answer 3

Thought I'd add a cautionary tale about removing outliers:

Remember the problem with the hole in the polar ozone layer? There was a satellite that was put in orbit over the pole specifically to measure ozone concentration. For a few years the post-processed data from the satellite reported that the polar ozone was present at normal levels, even though other sources clearly showed that the ozone was missing. Finally someone went back to check the satellite software. It turned out that someone had written the code to check if the raw measurement was within an expected range about the typical historical level, and to assume that any measurement outside the range was just an instrument 'spike' (i.e an outlier), auto-correcting the value. Fortunately they had also recorded the raw measurements; on checking them they saw that the hole had been reported all along.

Answer 4

'Outlier' is a convenient term for collecting data together that doesn't fit what you expect your process to look like, in order to remove from the analysis.

I would suggest never (caveat later) removing outliers. My background is statistical process control, so often deal with large volumes of automatically generated time-series data which is processed using a run chart / moving box plot / etc. depending on the data and distribution.

The thing with outliers is that they will always provide information about your 'process'. Often what you are thinking of as one process is actually many processes and it is far more complex than you give it credit for.

Using the example in your question, I would suggest there could be a number of 'processes'. there will be variation due to ...

• samples taken by one conductance device
• samples taken between conductance devices
• when the subject removed a probe
• when the subject moved
• differences within one subject's skin across their body or between different sampling days (hair, moisture, oil, etc)
• differences between subjects
• the training of the person taking the measurements and variations between staff

All of these processes will produce extra variation in the data and will probably move the mean and change the shape of the distribution. Many of these you won't be able to separate into distinct processes.

So going to the idea of removing data points as 'outliers' ... I would only remove data points, when I can definitely attribute them to a particular 'process' that I want to not include in my analysis. You then need to ensure that the reasons for non inclusion are recorded as part of your analysis, so it is obvious. Don't assume attribution, that's the key thing about taking extra notes through observation during your data collection.

I would challenge your statement 'because most of them are errors anyway', as they are not errors, but just part of a different process that you have identified within your measurements as being different.

In your example, I think it is reasonable to exclude data points that you can attribute to a separate process that you don't want to analyse.

Answer 5

If you are removing outliers the, in most situations you need to document that you're doing so and why. If this is for a scientific paper, or for regulatory purposes, this could result in having your final statistics discounted and/or rejected.

The better solution is to identify when you think you're getting bad data (e.g. when people pull wires), then identify when people are pulling wires, and pull the data for that reason. This will probably also result in some 'good' data points being dropped, but you now have a 'real' reason to tag and discount those data points at the collection end rather than at the analysis end. As long as you do that cleanly and transparently, it's far more likely to be acceptable to third parties. If you remove data points related to pulled wires, and you still get outliers, then the probable conclusion is that the pulled wires are not the (only) problem -- the further problem could be with your experiment design, or your theory.

One of the first experiments my mom had when returning to university to finish her BSc was one where students were given a 'bad' theory about how a process worked, and then told to run an experiment. Students who deleted or modified the resulting 'bad' data points failed the assignment. Those who correctly reported that their data was in disagreement with the results predicted by (the bad) theory, passed. The point of the assignment was to teach students not to 'fix' (falsify) their data when it wasn't what was expected.

Summary: if you're generating bad data, then fix your experiment (or your theory), not the data.

Answer 6

It's a moral dilemma for sure. On one hand, why should you let a few suspicious data points ruin your model's fit to the bulk of the data? On the other hand, deleting observations that don't agree with your model's concept of reality is a censorship of sorts. To @Egon's point, those outliers could be trying to tell you something about that reality.

In a presentation from statistician Steve MacEachern, he defined outliers as being "[not representative of the phenomenon under study.]" Under that viewpoint, if you feel that these suspicious data points are not representative of the skin conductance phenomenon you are trying to study, maybe they don't belong in the analysis. Or if they're allowed to stay, a method should be used that limits their influence. In that same presentation MacEachern gave examples of robust methods, and I remember that, in those few examples, the classical methods with the outliers removed always agreed with the robust analyses with the outliers still included. Personally, I tend to work with the classical techniques I'm most comfortable with and live with the moral uncertainty of outlier deletion.

Answer 7

My answer aligns with the majority: Do not remove outliers unless you are certain they are erroneous. What I add is:

• A brief overview of published papers on this topic (those that I am aware of and primarily those published in psychology. There are many more).
• Based on that, an answer to the question: What method can be used instead of removing outliers when one knows that there are many incorrect data points but not which ones are incorrect?

Overview

It is well-documented that the removal of outliers invalidates statistical results. Wilcox 1998 states: "This approach fails [removing outliers before a standard analysis] because it results in using the wrong standard error. Briefly, if extreme values are thrown out, the remaining observations are no longer independent, so conventional methods for deriving expressions for standard errors no longer apply." For a more detailed explanation, see the paper. Bakker et al. (2014) demonstrated one of the effects of this: substantially inflated type I error rates. Recently, Andre (2022) argued that this is only a problem when the model/hypothesis is considered for removing outliers. To provide a concrete example: He stated that while removing outliers within a group is problematic due to the invalid standard errors, removing outliers across groups is valid. More recently, Karch (2023)(disclaimer: that's me) demonstrated that removing outliers across groups is equally problematic: Among other things, if there are group differences, it almost always invalidates confidence intervals and parameter estimates.

What can be used with noisy data?

All the papers cited so far recommend robust methods for handling noisy data (as suggested in some answers). Importantly, contrary to what is claimed in other answers, robust methods do not always yield the same results as outlier removal + standard methods. Briefly, robust methods use the correct standard errors, while outlier removal + standard methods do not (refer to Wilcox for details).

For the situation the original question asks about (comparing two groups), either the Yuen-Welch test or the Brunner-Munzel test and their corresponding confidence intervals seem like they could be applicable. The Yuen-Welch test is essentially the robust version of Welch's t-test. It's important to note that it considers trimmed means instead of normal means, which can be very different for asymmetric distributions (see example by AdamO). Brunner-Munzel's test is essentially the robust alternative to the Wilcoxon-Mann-Whitney test (see https://stats.stackexchange.com/a/579604/30495). Both tests are readily available in R (see https://rdrr.io/cran/WRS2/man/yuen.html, and https://cran.r-project.org/web/packages/brunnermunzel/index.html)

Answer 8

If I conduct a random sample of 100 people, and one of those people happens to be Bill Gates, then as far as I can tell, Bill Gates is representative of 1/100th of the population.

A trimmed mean tells me the average lottery earnings is $0.

Answer 9

Of course you should remove the outliers, as by definition they do not follow the distribution under scrutiny and are a parasitic phenomenon.

The real question is "how can I reliably detect the outliers" !