Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 3 down vote favorite
2

I need advice on what is a a reasonable number of cases to be deleted as outliers.

I have conducted an outliers analysis to identify univariate and multivariate outliers from my dataset. Alltogether 30% of the data was classified as outliers.

If I delete all of these outliers, my results appear to improve. Also, after deleting the outliers my sample size is still good (i.e., n=300).

  • Is it reasonable to delete all the cases classified as outliers?
share|improve this question
6  
Why do you think that you need to remove the outliers? Do you have reason to believe that they are "bad" data, i.e. data entry errors, etc? In general, 0 is a reasonable number of outliers to remove. – Aniko Sep 14 '11 at 12:48
2  
Good point, @Aniko. Without further information demonstrating that an "outlier" is mistaken or irrelevant, 0 is the only defensible number of outliers to remove. However, it's possible (and usually a good idea) to conduct analyses both with and without the outliers to assess how much the outliers influence the results. – whuber♦ Sep 14 '11 at 16:04

1 Answer

up vote 11 down vote

I would be more than suspicious, if someone told me that 30% of my sample are outliers ...

Rather than blindly trusting a canned routine I would carefully analyze the data and try to find out why an outlier is an outlier. Is it a "bug" or a "feature"? Is it measurement error? Does your sample cover different sub-populations (mixture)?

Moreover, the detection of outliers involves the more or less arbitrary definition of a threshold, which separates "good" and "bad". You should assess if these thresholds are sensible. It could thus be a good idea to move the goalposts and to see what happens.

Also note that rather than dropping observations, you could use robust statistical techniques if you are concerned about outliers.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.