5
$\begingroup$

Background:
There are often some values among sampling set that appear not closely compatible with the rest. They would be called as extreme values or simply outliers. Dealing with outliers has been always a matter of challenge. There are some approaches to solve the problem of the existence of outliers:

  • moving them to a separated set
  • replacing them with nearest values from non-outlier set
  • ...

Question:
What is the most recommended method(s) to deal with outliers? (with details and an example)

$\endgroup$
  • $\begingroup$ Sometimes the outlier are where you should drill for oil. $\endgroup$ – Michael Hardy Dec 16 '13 at 23:02
8
$\begingroup$

I've recommended two methods in the past. They depend on the nature of the data in a general sense.

If the outliers are part of a well known distribution of data with a well known problem with outliers then, if others haven't done it already, analyze the distribution with and without outliers, using a variety of ways of handling them, and see what happens. You're going to be dealing with this data a lot. You might as well understand an outlier problem. For example, Ratcliff has a nice little paper on reaction times that you might look at as an example. If there are papers like that for your example then read them.

If the outliers are from a data set that is relatively unique then analyze them for your specific situation. Analyze both with and without them, and perhaps with a replacement alternative, if you have a reason for one, and report your results of this assessment.

So, in short, analyze and document. That's the best thing to do.

I should make it clear that an outlier needs to be defined relatively independently of the statistical distribution (in extent, not necessarily shape). For example, with reaction times you may define short outliers as those that aren't really reactions to the stimulus but instead, anticipations. Long ones might have a similar definition in that they are not reactions to the stimulus onset but something else (with the something else being potentially a variety of things). Going through and finding that 3% of your data points were more than 2 SDs away from the mean does not demonstrate that you have a small amount of outliers. On the contrary, it suggests you have no outliers and should keep them all.

$\endgroup$
  • $\begingroup$ After a quick review I would say that the paper is a good work (I recommend too). It is about 23 pages so requires time. I will be back then so. $\endgroup$ – Developer Oct 8 '11 at 15:33
  • $\begingroup$ I have discussed my work and the work of Martin and others on using influence functions to detect outliers. Influence functions tell you the effect of the outlier on an estimate of a parameter, basically telling you the difference between the estimate with the outlier in and the outlier taken out. This can be a shortcut method to do what John is suggesting. $\endgroup$ – Michael Chernick Jul 25 '12 at 22:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.