How to deal with outliers? Background:
There are often some values among sampling set that appear not closely compatible with the rest. They would be called as extreme values or simply outliers. Dealing with outliers has been always a matter of challenge. There are some approaches to solve the problem of the existence of outliers:  


*

*moving them to a separated set

*replacing them with nearest values from non-outlier set

*...


Question:
What is the most recommended method(s) to deal with outliers? (with details and an example)
 A: I've recommended two methods in the past.  They depend on the nature of the data in a general sense.  
If the outliers are part of a well known distribution of data with a well known problem with outliers then, if others haven't done it already, analyze the distribution with and without outliers, using a variety of ways of handling them, and see what happens.  You're going to be dealing with this data a lot.  You might as well understand an outlier problem.  For example, Ratcliff has a nice little paper on reaction times that you might look at as an example.  If there are papers like that for your example then read them.
If the outliers are from a data set that is relatively unique then analyze them for your specific situation.  Analyze both with and without them, and perhaps with a replacement alternative, if you have a reason for one, and report your results of this assessment.
So, in short, analyze and document.  That's the best thing to do.
I should make it clear that an outlier needs to be defined relatively independently of the statistical distribution (in extent, not necessarily shape). For example, with reaction times you may define short outliers as those that aren't really reactions to the stimulus but instead, anticipations. Long ones might have a similar definition in that they are not reactions to the stimulus onset but something else (with the something else being potentially a variety of things). Going through and finding that 3% of your data points were more than 2 SDs away from the mean does not demonstrate that you have a small amount of outliers. On the contrary, it suggests you have no outliers and should keep them all.
