Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I looked for a way to remove outliers from a dataset and I found this question.

In some of the comments and answers to this question, however, people mentioned that it is bad practice to remove outliers from the data.

In my dataset I have several outliers that very likely are just due to measurement errors. Even if some of them are not, I have no way of checking it case by case, because there are too many data points. Is it statistically valid than just to remove the outliers? Or, if not, what could be another solution?

If I just leave those points there, they influence e.g. the mean in a way that does not reflect reality (because most of them are errors anyway).

EDIT: I am working with skin conductance data. Most of the extreme values are due to artifacts like somebody pulling on the wires.

EDIT2: My main interest in analyzing the data is to determine if there is a difference between two groups

share|improve this question

migrated from stackoverflow.com Mar 8 at 13:02

This question came from our site for professional and enthusiast programmers.

2  
And what do you want to do? Data summary? Predictive analysis? Data visualization? Proving that there is (no) significant difference between two groups? As with all data cleaning, there is no general answer. – Piotr Migdal Mar 8 at 14:04
4  
3  
I'm an engineer who works with lots of statistics. That was a disclaimer and a confession that means I have to deliver products. We are only allowed to remove fully attributed "bad" points. Can you prove it was from someone pulling a wire? If you get several intentional measures, you can bound and cluster there. Then you can split the data on the cluster (pull vs non-pull) and it isn't about outliers anymore. If you can't prove what the root cause is, you must (must) retain it. It speaks to variation, and that is a big chunk of analysis. You can't get rid of it if you don't like it. – EngrStudent Mar 8 at 17:26
2  
I think you start by the wrong end. The first question is how do you identify the outliers in the first place? – user603 Mar 8 at 20:44
3  
Rather than arbitrary removal of arbitrarily identified outliers, you may be better to consider something like "since I have contamination from issues such as people pulling on wires, what methodologies can I use which are not badly affected by such contamination?" – Glen_b Mar 8 at 23:52

I do not recommend excluding any outlier in the main analysis (unless you are really positive they are mistaken). You can do it in a sensitivity analysis, though, and compare the results of the two analyses. In science, often you discover new stuff precisely when focusing on such outliers.

To further elaborate, just think about the seminal Fleming's discovery of penicillin, based on the accidental contamination of his experiments with a mold:

http://www.abpischools.org.uk/page/modules/infectiousdiseases_timeline/timeline6.cfm?coSiteNavigation_allTopic=1

Looking at the near past or present, outlier detection is often used to guide innovation in biomedical sciences. See for instance the following articles (with some suitable R codes):

http://www.la-press.com/a-comparison-of-methods-for-data-driven-cancer-outlier-discovery-and-a-article-a2599-abstract?article_id=2599

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3394880/

http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0102678

Finally, if you have reasonable grounds to exclude some data, you may do it, preferably in a sensitivity analysis, and not in the primary one. For instance you could exclude all values which are not biologically plausible (such as a temperature of 48 degrees Celsius in a septic patient). Similarly, you could exclude all first and last measurements for any given patient, to minimize movement artifacts. Take notice however that if you do this post-hoc (not based on a pre-specified criteria), this risks amounting to data massaging.

share|improve this answer
4  
Agree, but I find this answer somehow to brief to upvote it. Maybe you could provide a worked example, or show why and how new stuff can be discovered when focusing on outliers? This may be not that obvious at a first sight. – Tim Mar 9 at 9:35

One option is to exclude outliers, but IMHO that is something you should only do if you can argue (with almost certainty) why such points are invalid (e.g. measurement equipment broke down, measurement method was unreliable for some reason, ...). E.g. in frequency domain measurements, DC is often discarded since many different terms contribute to DC, quite often unrelated to the phenomenon you are trying to observe.

The problem with removing outliers, is that to determine which points are outliers, you need to have a good model of what is or is not "good data". If you are unsure about the model (which factors should be included, what structure does the model have, what are the assumptions of the noise, ...), then you cannot be sure about your outliers. Those outliers might just be samples that are trying to tell you that your model is wrong. In other words: removing outliers will reinforce your (incorrect!) model, instead of allowing you to obtain new insights!

Another option, is to use robust statistics. E.g. the mean and standard deviation are sensitive to outliers, other metrics of "location" and "spread" are more robust. E.g. instead of the mean, use the median. Instead of standard deviation, use inter-quartile range. Instead of standard least-squares regression, you could use robust regression. All those robust methods de-emphasize the outliers in one way or another.

share|improve this answer
3  
Great answer. Most people don't realize that not every technique is suited for every type of data. Concentrating on the mean for data ridden with outliers is one of the unfortunate results. The more wake-up calls they get, from answers like this, the better for everybody. – rumtscho Mar 9 at 9:59

Thought I'd add a cautionary tale about removing outliers:

Remember the problem with the hole in the polar ozone layer? There was a satellite that was put in orbit over the pole specifically to measure ozone concentration. For a few years the post-processed data from the satellite reported that the polar ozone was present at normal levels, even though other sources clearly showed that the ozone was missing. Finally someone went back to check the satellite software. It turned out that someone had written the code to check if the raw measurement was within an expected range about the typical historical level, and to assume that any measurement outside the range was just an instrument 'spike' (i.e an outlier), auto-correcting the value. Fortunately they had also recorded the raw measurements; on checking them they saw that the hole had been reported all along.

share|improve this answer
9  
It would be good to include a reference to the incident: Why hadn't they discovered the phenomenon earlier? Unfortunately, the TOMS data analysis software had been programmed to flag and set aside data points that deviated greatly from expected measurements and so the initial measurements that should have set off alarms were simply overlooked. In short, the TOMS team failed to detect the ozone depletion years earlier because it was much more severe than scientists expected. – Johnny Mar 9 at 2:06
    
This is a great story. and one much repeated, but to me math.uni-augsburg.de/stochastik/pukelsheim/1990c.pdf convincingly identifies it as a myth based on a misunderstanding. Note incidentally that as there are two poles, "the polar ozone layer" needs re-writing. – Nick Cox Mar 10 at 11:45
    
See also the authoritative account Christie. M. 2001. The Ozone Layer A Philosophy of Science Perspective. Cambridge: Cambridge U.P. – Nick Cox Mar 10 at 11:56

'Outlier' is a convenient term for collecting data together that doesn't fit what you expect your process to look like, in order to remove from the analysis.

I would suggest never (caveat later) removing outliers. My background is statistical process control, so often deal with large volumes of automatically generated time-series data which is processed using a run chart / moving box plot / etc. depending on the data and distribution.

The thing with outliers is that they will always provide information about your 'process'. Often what you are thinking of as one process is actually many processes and it is far more complex than you give it credit for.

Using the example in your question, I would suggest there could be a number of 'processes'. there will be variation due to ...

  • samples taken by one conductance device
  • samples taken between conductance devices
  • when the subject removed a probe
  • when the subject moved
  • differences within one subject's skin across their body or between different sampling days (hair, moisture, oil, etc)
  • differences between subjects
  • the training of the person taking the measurements and variations between staff

All of these processes will produce extra variation in the data and will probably move the mean and change the shape of the distribution. Many of these you won't be able to separate into distinct processes.

So going to the idea of removing data points as 'outliers' ... I would only remove data points, when I can definitely attribute them to a particular 'process' that I want to not include in my analysis. You then need to ensure that the reasons for non inclusion are recorded as part of your analysis, so it is obvious. Don't assume attribution, that's the key thing about taking extra notes through observation during your data collection.

I would challenge your statement 'because most of them are errors anyway', as they are not errors, but just part of a different process that you have identified within your measurements as being different.

In your example, I think it is reasonable to exclude data points that you can attribute to a separate process that you don't want to analyse.

share|improve this answer

It's a moral dilemma for sure. On one hand, why should you let a few suspicious data points ruin your model's fit to the bulk of the data? On the other hand, deleting observations that don't agree with your model's concept of reality is a censorship of sorts. To @Egon's point, those outliers could be trying to tell you something about that reality.

In a presentation from statistician Steve MacEachern, he defined outliers as being "[not representative of the phenomenon under study.]" Under that viewpoint, if you feel that these suspicious data points are not representative of the skin conductance phenomenon you are trying to study, maybe they don't belong in the analysis. Or if they're allowed to stay, a method should be used that limits their influence. In that same presentation MacEachern gave examples of robust methods, and I remember that, in those few examples, the classical methods with the outliers removed always agreed with the robust analyses with the outliers still included. Personally, I tend to work with the classical techniques I'm most comfortable with and live with the moral uncertainty of outlier deletion.

share|improve this answer
6  
In Box, Hunter & Hunter: "Statistics for Experimenters" they tell that, in the chemical industry, outliers often have led to new patents. Do you want to throw out your new patent? – kjetil b halvorsen Mar 8 at 20:40
1  
Nope, I don't want to miss out on any patents. But I also don't want to spin twelve cycles trying to get my model to accommodate "somebody pulling on the wires." That's almost certainly not the phenomenon under study. I do like the idea of outliers as opportunities, and one thing to be said for straightforward deletion is that at least the code will provide documentation of those deletions, whereas in robust methods the outliers just kind of coexist with the other points. – Ben Ogorek Mar 9 at 14:26
1  
You are right that the specific circumstances must be taken into consideration. What should not be done is apply some context-free "rules" for outlier rejection. There do not exist any such good rules. – kjetil b halvorsen Mar 9 at 14:36
    
My favorite point about the power of context is illustrated by the question, "Are Snickers bars healthy?" Well, if you've been lost in the woods for three days and you've just found a few on the ground, it turns out they're pretty healthy after all. I feel like the popular answers here are telling us, "Never eat a Snickers bar, unless you're absolutely sure you'll die if you don't." – Ben Ogorek Mar 13 at 21:07

If you are removing outliers the, in most situations you need to document that you're doing so and why. If this is for a scientific paper, or for regulatory purposes, this could result in having your final statistics discounted and/or rejected.

The better solution is to identify when you think you're getting bad data (e.g. when people pull wires), then identify when people are pulling wires, and pull the data for that reason. This will probably also result in some 'good' data points being dropped, but you now have a 'real' reason to tag and discount those data points at the collection end rather than at the analysis end. As long as you do that cleanly and transparently, it's far more likely to be acceptable to third parties. If you remove data points related to pulled wires, and you still get outliers, then the probable conclusion is that the pulled wires are not the (only) problem -- the further problem could be with your experiment design, or your theory.

One of the first experiments my mom had when returning to university to finish her BSc was one where students were given a 'bad' theory about how a process worked, and then told to run an experiment. Students who deleted or modified the resulting 'bad' data points failed the assignment. Those who correctly reported that their data was in disagreement with the results predicted by (the bad) theory, passed. The point of the assignment was to teach students not to 'fix' (falsify) their data when it wasn't what was expected.

Summary: if you're generating bad data, then fix your experiment, not the data.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.