I know there are tons of questions on CV.SE about outliers, but I didn't find a solution to my specific case.

I have a dataset that I'm analyzing where in order to achieve "good" results, the data must be restricted in several ways. When I began restricting, there were over 140,000 entries, and after cutting out all the "bad" entries, there's about 2,000 entries. The next step in my analysis is to cut out the outliers in the data. This is done based on the percentile of a previous draft of the dataset that has about 4,000 entries. A colleague of mine did a similar analysis and cut out any outliers that were below the 1st percentile and above the 99th percentile. It's worth noting that their dataset was much smaller than mine, so they couldn't restrict much more beyond that point without getting into issues of sample size.

My conundrum: The more outliers I cut out, the better my results get. I get truly terrible results when I only cut out outliers below the 1st and above the 99th percentile. I've gone all the way up to cutting out below the 10th and above the 90th percentile and my results are much more realistic. My question is how far can I go with this? What basis do I have for deciding where the cut-off is for restricting the data? My gut says that data between the 10th-90th percentile is too restricted already, but it's tempting to report it because the results are truly so much better.

  • $\begingroup$ I thought of Koenker's Dictionary of Received Ideas of Statistics entry for robustness: Burned with such intensity that, like Marxism-Leninism, only the ashes of the most pure remain.. Theoretically, you can go all the way to trimming 25% on each end. $\endgroup$ – user603 Jan 9 '15 at 23:41
  • $\begingroup$ "140,000 entries, and after cutting out all the "bad" entries, there's about 2,000 entries"... so wait... you throw out about 98.6% of your data ... and then begin trying to 'cut out outliers' ?? Can you explain more about this situation, with 98.6% 'bad data' ... and that's before you even consider outliers? $\endgroup$ – Glen_b Jan 10 '15 at 1:35
  • $\begingroup$ The entire database has 140,000 entries, but 94.1% of them are incomplete. This is ongoing research that started in the 1960s, so entries could be "finished" at any point. Once you cut out the 94% that aren't finished (and thus don't have any data), there are physical impossibilities in the dataset that need to be taken out. An example: one entry was "started" in 1985 and "finished" in 1969. This is data that cannot be used not because it gives bad results but because it is an obvious mistake. $\endgroup$ – Peter Jan 10 '15 at 2:37
  • $\begingroup$ Sounds like right-censoring. There are lots of techniques out there that let you use the entire dataset. For example, google for survival analysis to one important field that deals with right-censoring. Throwing away such a large proportion of your data is almost certainly a very bad idea; to what population can you hope to generalize your finding after you have done that? $\endgroup$ – Maarten Buis Jan 10 '15 at 8:39
  • $\begingroup$ Believe it or not, that's the correct way to conduct the analysis. Let me get more specific: this is tag and recapture data, so to analyze an entry you need two dates and two length measurements, one at release and one at recapture. The database includes entries of individuals that were tagged but have not yet been recaptured, so the 94.1% of incomplete entries means that this species has a recapture rate of 5.9%, which is actually high if you look in the literature. Once you get rid of individuals that weren't recaptured and then any obvious mistake entries, I'm still left with "ugly" data. $\endgroup$ – Peter Jan 10 '15 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.